E-commerce Site

````txt theme={null} # Playwright to Checkly — Check Suite Setup Prompt You are helping me add Checkly Playwright monitoring to an existing repository that already contains Playwright tests. ## Goal Set up the smallest possible working Checkly Playwright Check Suite from this repository, prove it works with `npx checkly test --record`, and prepare it for deployment with `npx checkly deploy`. ## Success Criteria 1. Create or update the minimal files needed so this repo has a valid `checkly.config.ts`. 2. Reuse my existing `playwright.config.ts`. 3. Select only a small subset of tests that are most suitable for synthetic monitoring. Do NOT try to cover the full codebase. 4. Make sure the selected checks are fast, stable, and understandable. 5. Let me choose the monitoring frequency and locations. 6. Ask the user for their alert notification email and/or Slack App destination, then create at least one alert channel in code and assign it to all checks. 7. Get me to a point where: * `npx checkly test --record` succeeds * `npx checkly deploy` is ready to run successfully 8. Explain what you changed and why. **You are NOT done until `npx checkly deploy` succeeds.** ## Important Constraints * Start small: prefer 1 check suite for 1 critical user flow. * Do not migrate the whole Playwright suite. * Avoid brittle or slow tests. * Prefer existing tagged tests or existing Playwright projects when available. * If there are no good tags/projects yet, choose the smallest viable test file or test group and propose the lightest-touch way to isolate it. You can use a custom `testCommand:` specifying exactly the file, grep or reverse grep that makes sense for Checkly to run. * Keep the monitoring setup easy to extend later. --- ## Checkly-Specific Requirements * Use `checkly.config.ts` with `defineConfig`. * Point `checks.playwrightConfigPath` to my existing Playwright config. * Add at least one entry to `checks.playwrightChecks`. * Every Playwright Check Suite entry must have `name` and `logicalId`. * Use `pwProjects` and/or `pwTags` when that is the cleanest way to select tests. * Configure `frequency` and `locations` for the check suite based on my choices. * If needed, set `cli.runLocation` to one of the chosen locations for local validation. * If this repo uses TypeScript for config/tests, make sure `checkly` is installed. * Only customize `installCommand` or `testCommand` if the repo layout or package manager requires it. ### Alert Channel Setup **Always create at least one alert channel in `checkly.config.ts` and assign it to all checks.** **Before generating the config, you MUST ask the user for at least one of the following:** - **Email alerts:** Ask for their alert notification email address, then create an `EmailAlertChannel` with it. - **Slack App alerts:** Ask whether the Checkly Slack App is installed for their Slack workspace, then ask for the Slack `#channel` names or `@user` handles to notify and create a `SlackAppAlertChannel` with them. - **Legacy Slack webhook alerts:** Only ask for a Slack incoming webhook URL if the user explicitly needs to preserve or import an existing deprecated webhook-based Slack alert channel. Do not ask users to create a new Slack webhook for a new monitoring setup. The user can provide one or both. At minimum, one must be configured. **Do NOT use placeholder values like `alerts@example.com` or `` — always use the real values the user gives you.** If the user explicitly tells you they already have an alert channel configured in the Checkly UI and want to reuse it, use `EmailAlertChannel.fromId()` or `SlackAppAlertChannel.fromId()` instead. In that case, ask them for the numeric alert channel ID from the Checkly UI. ### checkly.config.ts — Exact Pattern This is the ONLY file structure you should generate. All checks are defined inline in `playwrightChecks`. No separate check files, no `__checks__/` directory. ```typescript import { defineConfig } from 'checkly'; import { EmailAlertChannel, SlackAppAlertChannel, Frequency } from 'checkly/constructs'; // Email alert channel — remove this block if user does not provide an email const emailAlert = new EmailAlertChannel('default-email-alert', { address: 'THE_ACTUAL_EMAIL_THE_USER_GAVE_YOU', sendFailure: true, sendRecovery: true, sendDegraded: false, }); // Slack App alert channel — remove this block if user does not provide Slack App destinations const slackAlert = new SlackAppAlertChannel('default-slack-alert', { slackChannels: ['THE_ACTUAL_SLACK_CHANNEL_OR_HANDLE_THE_USER_GAVE_YOU'], sendFailure: true, sendRecovery: true, sendDegraded: false, }); export default defineConfig({ projectName: 'my-checkly-project', logicalId: 'my-checkly-project', repoUrl: '', checks: { // Point to the EXISTING Playwright config — do not create a new one playwrightConfigPath: './playwright.config.ts', // Monitoring locations locations: ['eu-west-1', 'us-east-1'], // Alerting declared in code — include whichever channels are configured alertChannels: [emailAlert, slackAlert], playwrightChecks: [ { name: 'Critical happy path', logicalId: 'critical-happy-path', pwProjects: ['checkly'], frequency: Frequency.EVERY_10M, }, ], }, cli: { runLocation: 'eu-west-1', retries: 0, }, }); ``` **Import rules:** * `defineConfig` from `'checkly'` * `Frequency`, `EmailAlertChannel`, and `SlackAppAlertChannel` from `'checkly/constructs'` (NOT from `'checkly'`) **Test selection — four strategies:** 1. **`pwProjects`** — reference Playwright projects by name. Most common. The project name in `playwright.config.ts` MUST exactly match the string in `pwProjects`. 2. **`pwTags`** — filter by Playwright `@tags` in test titles/annotations. Useful when you want to select tests across projects without creating a new project. 3. **Project dependencies** — for auth flows, reference a Playwright project that has `dependencies` on a setup project in the Playwright config. 4. **`testCommand`** — specify a working Playwright test command targeting a single spec, grep, or reverse grep option. **Do NOT use any of the following — they are wrong for Playwright Check Suites:** * `checkMatch` — not needed; checks are inline in `playwrightChecks` * `__checks__/` directory or separate `.check.ts` files * `runtimeId` — Playwright Check Suites auto-detect the runtime from `package.json` * `playwrightConfig` block in `defineConfig` — that is for Browser Checks, a different Checkly product ### Playwright Config Changes You will likely need to add a new Playwright project to the existing `playwright.config.ts` so that `pwProjects` has something to reference. You MUST also make Checkly-specific behavior obvious in `playwright.config.ts` using `CHECKLY`. When `process.env.CHECKLY` is set: * `retries` MUST be `2` * traces MUST be enabled * if a valid public monitoring URL is provided, use that URL instead of relying on the local `webServer` When `process.env.CHECKLY` is not set: * preserve existing local behavior as much as possible * do not unnecessarily slow down local or CI runs **Example — adding a `checkly` project and obvious `CHECKLY` behavior:** ```typescript const isCheckly = !!process.env.CHECKLY; // Add this to the projects array in playwright.config.ts { name: 'checkly', use: { ...devices['Desktop Chrome'], baseURL: isCheckly && process.env.PUBLIC_MONITOR_URL ? process.env.PUBLIC_MONITOR_URL : 'https://staging.example.com', }, }, export default defineConfig({ retries: isCheckly ? 2 : 0, use: { trace: isCheckly ? 'on' : 'retain-on-failure', }, }); ``` **Rules for modifying `playwright.config.ts`:** * Prefer additive changes. * Do NOT modify existing projects or their defaults unless required. * If possible, add a dedicated `checkly` project. * If the config already has a project that works for monitoring (e.g. an existing `chromium` project), you can reference it directly in `pwProjects` without creating a new one. * Ensure `devices` is imported from `@playwright/test` if not already. * Preserve existing local developer behavior as much as possible outside Checkly runs. --- ## Environment and Runtime Guidance * Preserve existing Playwright behavior as much as possible. * If the tests need environment variables, identify them clearly. * Use Checkly environment variables where helpful, especially `CHECKLY`, to distinguish Checkly runs from local runs. * Do not invent secrets. Instead, leave placeholders and tell me exactly what I must set. ## Timeout and Reliability Guidance * Optimize for a short-running check suite. * Avoid hard waits and unnecessarily long flows. * If a test is flaky, either avoid it or use the lightest reasonable retry/timeout adjustment. * Keep in mind that Playwright Check Suites can run up to 30 minutes, but we want something much shorter and better suited for alerting. --- ## URL Strategy Before choosing a `baseURL` for the monitoring project, understand what pages the tests actually navigate to. **Option A: Public URL** — Use when a deployed site serves ALL the pages the selected tests visit. Verify the public URL actually has every path the tests navigate to. For example, if tests call `page.goto('/test')` and the public site only serves `/` and `/docs`, the public URL will NOT work. If the public URL works, conditionally disable the `webServer` so local dev still works: ```typescript webServer: process.env.CHECKLY ? undefined : { /* keep existing webServer config exactly as-is */ }, ``` Only do this if you've verified the public URL serves ALL the test pages. If in doubt, keep the webServer. **Option B: Local webServer** — Use when test pages only exist on the dev server (test fixtures, demo apps, special endpoints). Keep the `webServer` enabled and use `http://localhost:` as `baseURL`. This is more common for libraries/frameworks with test harnesses. If a public URL is not provided, or does not make sense for the monitored flow: * keep the `webServer` * make sure the Playwright Check Suite uses an `include` block to bundle all files required to start the app/server remotely in Checkly * do not assume test files alone are sufficient * explicitly include additional app/server/source/build/config files as needed, because Checkly includes test-related files by default **If the webServer has native dependencies** (`node-pty`, `sharp`, `canvas`, `bcrypt`, `sqlite3` etc. imported at the top level of the server entry file), the project is NOT a Check Suite candidate. Route to the [Reporter Fallback](#reporter-fallback) instead. **Hardcoded localhost URLs in test files:** If selected test files have hardcoded `localhost` or `127.0.0.1` URLs and you're using a public URL (Option A), replace them with a `BASE` variable: ```typescript const BASE = process.env.CHECKLY ? 'https://staging.example.com' : 'http://localhost:'; await request.get(`${BASE}/api/health`); ``` Only do this for files that have hardcoded URLs. Don't touch files that use relative paths — those work via `baseURL`. --- ## Reporter Fallback Use the Reporter path ONLY when Check Suites truly cannot work: * Tests don't navigate to any page (no `page.goto()`, no `webServer`) * ALL test files have hard blockers (browser extensions, hardware dependencies, native deps in webServer) **Before choosing this path, verify:** Does any Playwright config have a `webServer`? Do the tests use `page.goto()`? Is there a public URL? If any of these could make Check Suites work, go back and try. ```bash npm install --save-dev @checkly/playwright-reporter ``` ```typescript import { createChecklyReporter } from '@checkly/playwright-reporter'; reporter: [ // ... existing reporters stay as-is ... createChecklyReporter({ sessionName: process.env.CI ? `CI — ${process.env.GITHUB_REF_NAME ?? 'main'} @ ${(process.env.GITHUB_SHA ?? '').slice(0, 7)}` : `Local — ${new Date().toISOString().slice(0, 16)}`, }), ], ``` If the config has a single reporter string (e.g., `reporter: 'list'`), convert it to array format first. Optionally add asset capture if missing: ```typescript use: { screenshot: 'only-on-failure', video: 'retain-on-failure', trace: 'on', }, ``` --- ## What I Want From You 1. Inspect the repository structure and identify: * the Playwright config file * whether `webServer` is used * candidate smoke/critical tests * any existing tags or Playwright projects * any environment variables the selected tests need 2. Recommend the best first monitoring target and briefly justify it. 3. Implement the minimal Checkly setup. 4. Show me the exact `checkly.config.ts` you create or the exact diff. 5. Show me the exact `playwright.config.ts` changes needed to: * enable traces when `CHECKLY` is set * set retries to `2` when `CHECKLY` is set * use a public URL for Checkly runs when appropriate * otherwise preserve the local `webServer` path 6. Ensure the final config creates and assigns at least one alert channel (Email and/or Slack). 7. If no public URL is used, show the exact `include` entries needed. 8. Show me any test edits you make, but keep them minimal. 9. Give me the exact commands to run in order. 10. Tell me what values I need to choose for: * frequency * locations * environment variables / secrets * alert notification email and/or Slack App destination 11. After implementation, provide a short final checklist confirming I should be able to run: * `npm install` or equivalent * `npx checkly test --record` * `npx checkly deploy` ## Decision Rules * Prefer one check suite over many. * Prefer one critical happy-path flow over broad coverage. * Prefer existing `pwProjects` or `pwTags` over restructuring tests. * If multiple options exist, choose the one with the fewest code changes. * If something is ambiguous, make the most conservative choice that gets to a working setup quickly. --- ## User-Provided Monitoring Settings Use these defaults unless the repo makes them invalid. **For values marked "ASK USER", you MUST ask the user before proceeding — do not use placeholders or example values.** * Desired frequency: `EVERY_10M` * Desired locations: `['eu-west-1', 'us-east-1']` * Target environment/base URL: `https://staging.example.com` * Extra environment variables to wire in: `none` * Alert notification email: **ASK USER** — ask for their real email address * Slack App installed: **ASK USER** — ask whether the Checkly Slack App is installed for their Slack workspace (optional if email is provided) * Slack App destinations: **ASK USER** — ask for Slack `#channel` names or `@user` handles (optional if email is provided) ## Reference Documentation Use these Checkly docs as the source of truth for the implementation: * [Quickstart — Playwright Check](https://www.checklyhq.com/docs/detect/synthetic-monitoring/playwright-checks/quickstart/) * [Test Organization](https://www.checklyhq.com/docs/detect/synthetic-monitoring/playwright-checks/test-organization/) * [Configuration](https://www.checklyhq.com/docs/detect/synthetic-monitoring/playwright-checks/configuration/) * [Environment Variables](https://www.checklyhq.com/docs/detect/synthetic-monitoring/playwright-checks/environment-variables/) * [Timeouts](https://www.checklyhq.com/docs/detect/synthetic-monitoring/playwright-checks/timeouts/) * [Email Alert Channel](https://www.checklyhq.com/docs/constructs/email-alert-channel/) * [Slack App Alert Channel](https://www.checklyhq.com/docs/constructs/slack-app-alert-channel/) --- Now inspect this repository and implement the smallest viable working Checkly Playwright Check Suite. ````

Welcome to Our E-commerce Site

## 3. Monitor network traffic with `waitForResponse` Playwright has built-in functions to wait for an event before continuing with execution. Waiting for the network is a good example; you can wait for any request or response before continuing. This example waits for any response matching the given pattern: ```ts network-waiting.spec.ts theme={null} // Attach network event listener const responsePromise = page.waitForResponse('site.example/modalBackend'); // Trigger network request await page.getByText('trigger response').click(); // Wait until the response arrived const response = await responsePromise; ``` We can now make assertions about this response when it arrives. Note that `waitForResponse()` isn't limited to URL matching, as it also accepts patterns. But sometimes, there will be multiple requests to a URL. For example, when polling to see if a transaction was successful, and we probably want to evaluate many responses and only continue the test when the transaction came through. In these cases, `waitForResponse` lets us access and wait for response details, too. ```ts advanced-network-waiting.spec.ts theme={null} // Attach network event listener and // wait for PUT requests receiving a 200 status code const responsePromise = page.waitForResponse(response => response.url() === 'site.example/modalBackend' && response.status() === 200 && response.request().method() === 'PUT' ); // Trigger network request await page.getByText('trigger response').click(); // Wait until the response arrived const response = await responsePromise; ``` A note on code reading: for myself, and many other users on Stack Overflow, it's quite difficult to get the promise structure here right: there is no await in the `responsePromise` definition, rather `await` is used only when we create a variable with the return from `responsePromise`. See response body waiting in action as demonstrated by Stefan here: ## 4. Implement your own auto-waiting and retry mechanism If there's no way to change the application code and no obvious network request to wait for, you could also make your tests pass by implementing you own retry mechanisms. If we consider the modal example, you can try to repeat your actions until they pass. If your first button click doesn't yield the expected results (showing the modal), try it again until it does. Playwright's `toPass` assertion allows you to implement precisely this functionality. ```ts retry-with-topass.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('site.example/modalLoader.html'); // Retry this code block until actions and assertions succeed // or timeouts are hit and the test fails await expect(async () => { await page.getByRole('button', { name: 'Toolbar 1' }).click(); await page.locator('#modal1').getByText('Close').click(); }).toPass() }); ``` By placing the initial instructions for opening and closing the modal in the `toPass` method, Playwright will retry these actions until they pass. If the modal isn't opening after the first click, it will try again until it is able to close it again or timeout and fail your test entirely. This approach won't fix the application's UI/UX issues but allows you to "work around" them in your end-to-end tests. Stefan explains it in detail on YouTube if you want to learn more about this approach. ## Conclusions In summary, while it may be tempting to use hard waits in your Playwright tests, they often create more problems than they solve. They can make your tests flaky, slow, and harder to maintain. Instead, Playwright provides many tools to handle dynamic scenarios: 1. **Rely on auto-waiting** whenever possible. 2. **Modify the page structure** to better reflect readiness if you have access. 3. **Use degraded states** for slow responses, rather than failing the test. 4. **Monitor network traffic** with `waitForResponse` to sync with backend events. 5. **Implement specific auto-waiting mechanisms** with `toPass`. Choosing smarter alternatives to hard waits will make your tests more robust, maintainable, and less flaky while accurately reflecting real-world user interactions. Win win! # In-Depth Monitoring With Checkly and Claude Code Source: https://www.checklyhq.com/docs/guides/claude-code-monitoring By integrating AI into your development workflow, you can generate valid Checkly constructs—including URL monitors and Playwright-based browser checks—with minimal effort. AI-assisted coding promises to massively upgrade developer productivity, and with Checkly's model of [Monitoring as Code](https://www.checklyhq.com/blog/monitoring-as-code-in-your-sdlc/), you can create monitoring coverage for all your services in minutes instead of days. This guide will show you how to create an AI development environment that lets you create and deploy several types of Checkly monitoring. ## Setup If you haven't already, start by [installing the Checkly CLI](https://www.checklyhq.com/docs/cli/installation/), then create a repository that will be the 'project' that contains all your Checkly monitors managed as code. If you don't already have a project, create one with: ```bash theme={null} npx checkly init ``` Next, install [Claude Code](https://www.anthropic.com/claude-code) globally with ```bash theme={null} npm install -g @anthropic-ai/claude-code ``` and from the root folder of your project, start Claude Code with the `claude` command. You'll be prompted to sign in to or sign up for an Anthropic account. ### Add the Playwright MCP You can give Claude Code access to a browser by adding the [Playwright MCP](https://github.com/microsoft/playwright-mcp) to your local configuration with: ```bash theme={null} claude mcp add playwright npx @playwright/mcp@latest ``` This will let us ask Claude Code to do things like crawling a particular website and interacting with pages. ## Give Claude Code Some Context Everything we write into the CLI interface is part of a 'prompt,' and by default, Claude Code will add a bit of information to what we submit as 'context' on that prompt. Ideally, our requests written into Claude Code would include context about our specific project: its structure, language, and even its purpose. Because of the `npm create` command we ran above, our folder already has a boilerplate version of a Checkly project. We can't feed in all of these files as context, but we can add a summary of our whole project as context by opening Claude Code and running [the slash command](https://docs.anthropic.com/en/docs/claude-code/slash-commands): ```{title="Prompt:"} theme={null} /init ``` Claude Code will scan the whole project, write a summary, and then store that summary in a local `CLAUDE.md` file. Claude Code will read that file and add it to our prompt every time it's run within this project from now on. Let's add a bit more to this context, we want test files that make sense as a [Checkly construct](/docs/constructs/overview). Let's add [Checkly's AI rules file](/docs/ai/overview) to our `CLAUDE.md` context file: ```bash theme={null} mkdir -p .claude && curl -o .claude/checkly.rules.md https://www.checklyhq.com/docs/ai/checkly.rules.md -L echo "- examine checkly.rules.md for code generation rules" >> .claude/CLAUDE.md ``` After this you'll need to exit and restart Claude Code for it to pick up changes to our context file. ## Create new URL Monitors We'll start with something easy. A URL monitor that will check an HTTP endpoint and alert if it doesn't receive the right status code. With the context set from the section above, try a command like: ```{title="Prompt:"} theme={null} Create a new URL monitor for the site https://danube-webshop.herokuapp.com/ that runs every five minutes and doesn't follow redirects. ``` After a few seconds, Claude Code will produce this: ```ts theme={null} import { Frequency, UrlMonitor, UrlAssertionBuilder } from 'checkly/constructs' new UrlMonitor('danube-web-shop-monitor', { frequency: Frequency.EVERY_5M, name: 'Danube Web Shop URL monitor', activated: true, request: { url: 'https://danube-web.shop/', skipSSL: false, followRedirects: false, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) ``` While we're writing this simplest of monitors, it's worth testing the limits of Claude Code's context for writing valid Checkly configuration. One thing we tested while writing this article was whether picking the wrong prompt would result in invalid Checkly construct code. The `Frequency` class here doesn't accept arbitrary values, so we wondered if the context provided in `CLAUDE.md` was enough for the tool to identify valid values for the check. We tried requesting a check that ran “every 17 seconds” and Claude Code prompted to run a `find` on the project to identify valid values for `frequency`. In the end, Claude Code did create valid code with this note in the process feed. ```{title="Reply from Claude Code:"} theme={null} Based on the search results, 17 seconds is NOT a valid frequency value for Checkly monitors. The closest valid option was selected: 20 seconds. ``` ## Create Playwright Synthetics Checks With Claude Code With our context set up, and a browser available via the Playwright MCP, we can create more complex Playwright scripts for browser monitoring of our site. With specific instructions and clear prompting, you can get consistent performance of Claude Code writing our Browser checks. ### 1. Start With a single test First you'll want to write a single check that does straightforward browsing and checking for page elements. With some context engineering, you can instruct Claude to navigate a page while recording the taken actions to generate a Playwright test. Read more about this approach and [extensive prompt guidelines on our blog](https://www.checklyhq.com/blog/generate-end-to-end-tests-with-ai-and-playwright/). ```{title="Prompt:"} theme={null} You are a Playwright test generator and an expert in TypeScript, Frontend development, and Playwright end-to-end testing. - You are given a scenario, and you need to generate a Checkly spec.ts test for it. - Navigate to the homepage https://danube-web.shop/ - Browse through multiple pages. - Assert the visibility of 1-2 page elements. ``` Claude will then open a browser using the Playwright MCP server, perform the actions, and create a short Playwright script that will look something like the following. ```ts {title="book-detail-visibility.spec.ts"} theme={null} import { test, expect } from '@playwright/test' test.describe('Danube WebShop Book Detail Page', () => { test('click book link and verify three elements are visible', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByText('Parry Hotter').click() await expect(page.getByRole('heading', { name: 'Parry Hotter' })).toBeVisible() await expect(page.getByRole('button', { name: 'Add to cart' })).toBeVisible() await expect(page.locator('img').first()).toBeVisible() }) }) ``` Lets make sure our script is valid by running this test: ```bash theme={null} npx checkly test book-detail ``` *Without an exact file path, the `checkly test` command will run any test that matches this pattern.* and making sure the test passes. ```bash theme={null} Bundling project resources... ✅ Running 1 checks in eu-west-1. __checks__/book-detail-visibility.spec.ts ✔ book-detail-visibility.spec.ts (7s) 1 passed, 1 total ``` *Like always with the `checkly test` command, this test isn't running from our workstation but rather from the Checkly's monitoring infrastructure.* Next we'll want to create a `.check.ts` file for this browser check, which sets some check-specific settings and assigns the check's logical ID. ```{title="Prompt:"} theme={null} Write a .check.ts file for book-detail-visibility that runs the check from a US and EU region and defines a linear retry strategy. ``` This should result in a complete `.check.ts` file generated by Claude, implementing the Checkly construct configuration. For better understanding of the configuration, you can ask for code comments: ```{title="Prompt:"} theme={null} Add comments to the check.ts file defining how each line configures the checkly construct. ``` The result is a nicely commented version of and a good tour of this construct: ```ts {title="book-detail-visibility.check.ts"} theme={null} // Import Checkly constructs for browser checks, alert escalation, and retry strategies import { Frequency, AlertEscalationBuilder, BrowserCheck, RetryStrategyBuilder } from 'checkly/constructs' // Create a new browser check with unique logical ID new BrowserCheck('book-detail-visibility-check-v1', { // Human-readable name displayed in Checkly dashboard name: 'Book Detail Page Visibility Check', // Configuration for the test code to execute code: { // Path to the Playwright test file relative to this check file entrypoint: './book-detail-visibility.spec.ts', }, // Whether the check is enabled and will run according to schedule activated: true, // Whether alerts are suppressed (false = alerts will be sent) muted: false, // Whether the check is expected to fail (false = failure will trigger alerts) shouldFail: false, // Geographic regions where the check will run locations: [ 'us-east-1', // US East (N. Virginia) 'eu-west-1', // EU West (Ireland) ], // How often the check runs frequency: Frequency.EVERY_30M, // Alert escalation policy defining when and how alerts are sent alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(1, { // Number of failed runs before sending alert (0 = immediate) amount: 2, // Time interval between escalation checks in minutes interval: 5, }, { // Whether percentage-based escalation is enabled enabled: false, // Percentage threshold for escalation (when enabled) percentage: 10, }), // Retry strategy configuration for failed checks retryStrategy: RetryStrategyBuilder.linearStrategy({ // Initial wait time before first retry in seconds baseBackoffSeconds: 30, // Maximum number of retry attempts maxRetries: 2, // Maximum total time for all retries in seconds maxDurationSeconds: 300, // Whether retries should use the same region as original check sameRegion: true, }), // Whether to run checks in multiple locations simultaneously runParallel: true, }) ``` Alternatively, you can deploy a Playwright check without a `.check.ts` configuration file, and the file name will be used as the logical ID. ### 2. Add our new Check to Claude's Context With this check created, let's add the following lines to your updated `CLAUDE.md` file: ```markdown theme={null} - When writing Playwright, don't set locators equal to const, rather just perform expect tests directly on locators - When writing a spec.ts file, don't use locators based on CSS class Oh that's smart. yeah I'll add something like: - Before creating more check files scan `__checks__` for more info and base setup ``` Pro tip: from the Claude Code prompt, just hit octothorp "#" to add to Claude's memory. Each entry will be a new line in `CLAUDE.md`. ### 3. Use Claude Code to create multiple Browser Checks Now that we've got a working test and, let's let Claude Code create another check for us: ```{title="Prompt:"} theme={null} Create a new browser check for https://danube-web.shop/ that clicks on a single button then checks the visibility of 10 different elements on the page. ``` As the prompt didn't specify which button to click, Claude Code will select one. In this case it picked a the signup button. The result is a nice clean test written to the spec, even the title is specific to the button that Claude Code decided to click: ```ts {title="signup-form-elements.spec.ts"} theme={null} import { test, expect } from '@playwright/test' test.describe('Danube WebShop Signup Form Elements', () => { test('click signup button and verify ten form elements are visible', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Sign up' }).click() await expect(page.getByRole('dialog')).toBeVisible() await expect(page.getByRole('textbox', { name: 'Name' }).first()).toBeVisible() await expect(page.getByRole('textbox', { name: 'Surname' })).toBeVisible() await expect(page.getByRole('textbox', { name: 'Email' })).toBeVisible() await expect(page.getByRole('textbox', { name: 'Password' })).toBeVisible() await expect(page.getByRole('textbox', { name: 'Company (optional)' })).toBeVisible() await expect(page.getByRole('radio', { name: 'Myself' })).toBeVisible() await expect(page.getByRole('radio', { name: 'My business' })).toBeVisible() await expect(page.getByRole('checkbox', { name: 'I would like to receive promotional emails' })).toBeVisible() await expect(page.getByRole('checkbox', { name: 'I have read and accept the privacy policy' })).toBeVisible() await expect(page.getByRole('button', { name: 'Register' })).toBeVisible() }) }) ``` Claude Code will confirm before creating the script file. If you've followed these tutorial steps in a single conversation with Claude Code, the tool will likely offer to `create signup-form-elements.check.ts` to configure the check. You can prompt for multiple checks at the same time. As you can imagine, this unlocks the creation of a large number of synthetic monitors relatively quickly. ```{title="Prompt:"} theme={null} Load the page https://danube-web.shop/ and create several separate .spec.ts checks, one for each page linked off of danube-web.shop. Each created check should check for multiple page elements. ``` If you haven't watched the responses from Claude Code already, it can be helpful to see how the tool breaks down our request: ```{title="Reply from Claude Code:"} theme={null} ⏺ Update Todos ⎿ ☐ Load danube-web.shop and identify all linked pages ☐ Explore each linked page to understand available elements ☐ Create .spec.ts files for each linked page with multiple element checks ``` Every file output from this command isn't included here for brevity, but Claude Code did create five new check scripts based on the five heading links at danube-web.shop. To test all these new checks quickly, run ```bash theme={null} npx checkly test ``` *With no file name argument, this will try to run everything in the `__checks__` folder* ```bash theme={null} 2 failed, 5 passed, 7 total ``` *If you've followed every step in this tutorial, you may have different numbers for the number of tests run here.* When running a number of generated tests, the only issue we ran into was a failure with the message: `Error: expect.toBeVisible: Error: strict mode violation: getByText('Crime & Thrillers') resolved to 2 elements:`. As the error message implies, the issue is that this locator: ```ts theme={null} await expect(page.getByText('Fantasy')).toBeVisible(); ``` Found multiple results on the page. There are a number of ways to fix this with Claude Code. We could prompt Claude Code with: ```{title="Prompt:"} theme={null} Revise locators in item-visibility.spec.ts to look at only the first result. ``` This would change every locator in the check, which may or may not be what you want! You could also paste in the error message to Claude and ask it to fix the script, or specify which locator you want to add `first()` to. Whatever route you take the result will be an edit to: ```ts theme={null} await expect(page.getByText('Fantasy').first()).toBeVisible(); ``` Once Claude Code added a single `.first()` to the checks written above, everything was passing. If you're having trouble reading the generated tests, take a look at [Checkly's Playwright documentation](https://www.checklyhq.com/learn/playwright/). ### 4. Going Further: Edit Multiple Checks and Configurations Updating checks is another use case that may be useful for larger projects. Try prompting Claude Code with something like: ```{title="Prompt:"} theme={null} Update every `.check.ts` file for a check that touches [danube-web.shop](http://danube-web.shop/), and change the frequency to every 30 minutes. ``` Not only do requests to update monitoring configuration work, sometimes they even produce sensible feedback and information about the checks that currently had no frequency setting: ```{title="Reply from Claude Code:"} theme={null} The url.check.ts and api.check.ts files don't have individual frequency settings - they inherit the default frequency from the project configuration in checkly.config.ts. ``` Once we're done creating new checks, we can test them with `npx checkly test` like we did above. Then it's time to run them all. ## Deploy With the Checkly CLI In our example, we want to deploy a couple of checks at once, so it's worth checking whether the project is properly set up before deployment. Many problems would have been revealed while running `test`, but if, for example, some checks were unintentionally removed from our project, we'll need to use `deploy` to detect that. Run the `deploy` command with the `-p` preview flag: ```bash theme={null} npx checkly deploy -p ``` This command won't deploy anything, just parse our project and show us what will be changed when we run the command without the preview flag. When you first run `deploy -p` you should just see a long list of created checks in the deployment command. For this tutorial, here's the preview of a more complex project state: ```bash theme={null} Bundling project resources... ✅ Create: BrowserCheck: __checks__/crime-category.spec.ts BrowserCheck: __checks__/fantasy-category.spec.ts BrowserCheck: __checks__/homepage-elements.spec.ts BrowserCheck: __checks__/navigation-menu.spec.ts BrowserCheck: __checks__/search-functionality.spec.ts BrowserCheck: book-detail-visibility-check-v1 BrowserCheck: signup-form-elements-check-v1 Delete: Check: danube-web-shop-monitor Update and Unchanged: BrowserCheck: __checks__/homepage.spec.ts ``` Let's discuss the output of this preview command in detail, as it's critical to understand how Checkly handles deployments from your project: * Checks that have no `.check.ts` config file are listed by their filename and will be named for their file name in the Checkly web UI. In general, it's better to include a config file for every check since it gives you explicit control of the check's logical ID. We've left out the config for five checks for demonstration purposes. * Checks with a config file are shown by their logical ID. * Checks previously deployed from this project that are not present in this deployment will be deleted. * Checks with a matching logical ID from a previous deployment will be listed as "Update and Unchanged". If you're getting strange results from a preview of `deploy` and you're not sure why, it might be time to get in touch with the Checkly team. [Join our Slack](https://checklyhq.com/slack) for deployment advice! Once we're happy with what will be deployed, it's time to run `npx checkly deploy` and create our checks! ## Conclusion AI-assisted coding tools like Claude Code can significantly accelerate the creation and deployment of monitoring checks with Checkly's Monitoring as Code approach. By integrating AI into your development workflow, you can generate valid Checkly constructs—including URL monitors and Playwright-based browser checks—with minimal effort. One of the superpowers that an AI workflow can unlock is the *mass updating of many files in your project.* Updating a large number of test files can go much faster with the help of Claude Code. While AI can handle much of the boilerplate work, human oversight remains essential for ensuring best practices, refining assertions, and maintaining high-quality monitoring scripts. By combining AI efficiency with developer expertise, teams can achieve comprehensive monitoring coverage in minutes rather than days. ### Further Reading Throughout this tutorial, we added a few lines to the CLAUDE.md context file. In all, those additions were (rewritten for clarity): ```markdown theme={null} ## Memories and Best Practices - Follow the rules in .claude/checkly.rules.md ### When writing checkly playwright tests - load the target page in a browser - don't set locators equal to const, rather just perform expect tests directly on locators - don't use locators based on CSS class ``` To learn more about Checkly's capabilities, explore these resources: * [Generating end-to-end tests with AI and Playwright MCP](https://www.checklyhq.com/blog/generate-end-to-end-tests-with-ai-and-playwright/) - Stefan's blog post makes a great starter for generalized Playwright development with an AI toolkit. * [Checkly CLI Documentation](/docs/cli/overview) – Learn how to manage checks as code right from your command line. * [Playwright Testing Guide](https://www.checklyhq.com/learn/playwright/) – Best practices for writing reliable browser checks. * [What is monitoring as code?](https://www.checklyhq.com/guides/monitoring-as-code/) — A comprehensive guide to infrastructure-as-code principles for monitoring. * [Checkly Community Slack](https://checklyhq.com/slack) – Join discussions and get support from the Checkly team and other users. For more tutorials and advanced use cases, visit the [Checkly YouTube channel](https://www.youtube.com/@ChecklyHQ), where you can see great tutorials like this guide to the Playwright MCP server: # Communicate User Feature Availability with Status Pages Source: https://www.checklyhq.com/docs/guides/communicate-availability Learn how Checkly status pages reflect actual user experience through synthetic monitoring, not arbitrary status indicators. Most status pages are disconnected from reality. They show green uptime bars based on server pings and health checks - metrics that tell you nothing about whether users can actually complete a purchase or log in. When infrastructure looks healthy but the checkout flow is broken, those green bars become meaningless. Checkly status pages go beyond reactive manual updates and infrastructure telemetry. They're powered by synthetic monitoring that simulates real user behavior, so when your status page shows "operational," it means users can actually complete their workflows. ## Where traditional status page setups fall short Traditional status pages suffer from a fundamental problem: **they communicate infrastructure health, not user experience**. Your servers might report healthy CPU usage while users can't log in because of [an incorrectly used React feature](https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-12-dashboard-and-api-outage/). Your database might show normal query times while users can't search for products. Infrastructure monitoring matters but only tells part of the story. The disconnect of green status bars and broken user experience erodes trust. Users learn to ignore status pages because they've been burned before by "all systems operational" banners during outages they're actively experiencing. Outages and bugs are unavoidable. Being transparent and honest about them is what matters and builds trust in your service. ## How Checkly status pages work [Checkly status pages](/docs/communicate/status-pages/overview) offer everything your current status page provider offers, plus integration with synthetic monitors that validate real user behavior. When you connect a [Playwright Check Suite](/docs/detect/synthetic-monitoring/playwright-checks/overview) or [Browser Check](/docs/detect/synthetic-monitoring/browser-checks/overview) that simulates a user logging in, adding items to cart, and completing checkout, your status page reflects whether that entire flow actually works. Following this approach, **your status page reflects what matters to your users.** Here's how the pieces fit together: 1. **Synthetic monitors validate behavior** - Playwright Check Suites and Browser Checks use Playwright to simulate user actions. These aren't simple ping tests or infrastructure checks; they're validations of your service's critical user flows in a real browser. 2. **Services represent user-facing capabilities** - You can define services like "Checkout" or "Login" that map to how users think about your application, not your internal architecture. 3. **Incident automation connects the dots** - When a check fails, it can automatically open an incident on the connected service. When the check recovers, the incident resolves. This means your status page shows what matters: **can users actually use your application?** ## Set up a status page backed by real synthetic monitoring ### Create services that match user expectations Services should reflect how users perceive your application. Users care about "Login" working, not whether your auth microservice cluster is healthy. Good service examples: * Website * User Login * Payments * Search Avoid internal naming like `Auth Service v2` or `Primary Database Cluster`. To create a service: 1. Navigate to **Services** under **Communicate** in the sidebar 2. Create a new service with a user-friendly name Services route showing multiple created services

Services route showing multiple created services

### Connect synthetic monitors to services This is where the real behavior validation happens. Each service can be connected to one or more monitors that validate its functionality. Incident automation is available on Communicate Team and Enterprise plans. [View pricing](https://checklyhq.com/pricing) 1. Open your Playwright Check Suite or Browser Check from the home dashboard 2. Click **Edit** in the check overview page 3. Click **Settings** and enable **Incident automation** 4. Fill in the incident name and initial status update 5. Select which service the incident should be opened on 6. Save your check Incident automation configuration of a Playwright Check Suite

Incident automation configuration of a Playwright Check Suite

### Create the status page 1. Go to **Status pages** under **Communicate** in the sidebar 2. Create a new status page 3. Enter a name for your page 4. Add cards and assign services to them. Group related services on the same card to show average uptime 5. Configure domain settings and your status page's appearance 6. Click **Create status page** Status page connecting to user experience services

Status page connecting to user experience services

Your status page now displays real-time availability based on actual user behavior validation. Created Checkly Status Page

### Automate everything with Monitoring as Code Checkly's [Monitoring as Code](/docs/guides/getting-started-with-monitoring-as-code) approach enables you to automate the entire flow of creating status pages, connecting services, and configuring checks. ```ts highlight={10-13,15-25,27-35,44} theme={null} import { Frequency, IncidentTrigger, PlaywrightCheck, StatusPage, StatusPageService, } from "checkly/constructs"; // 1. Create a new service to group checks and trigger incidents const searchService = new StatusPageService("search-service", { name: "Search Service", }) // 2. Create a new status page and connect the service new StatusPage("company-status", { name: "User Experience Status", url: "ux-status", cards: [ { name: "User Experience", services: [searchService], }, ], }) // 3. Configure your incident automation const searchIncidentTrigger: IncidentTrigger = { service: searchService, severity: "MINOR", name: "Search is down", description: "Some users experience issues with the product search. We're investigating.", notifySubscribers: true, } // 4. Assign your incident automations to checks and monitors new PlaywrightCheck("playwright-check-suite", { name: "Search Monitoring", playwrightConfigPath: "../playwright.config.ts", activated: true, pwProjects: ["Search Monitoring"], locations: ["us-east-1", "eu-west-1", "ap-southeast-2"], frequency: Frequency.EVERY_10M, triggerIncident: searchIncidentTrigger, }) ``` ## Why this approach works **A status page backed by synthetic monitoring builds trust because it tells the truth.** When users see "operational," they can trust that the application actually works. When there's an incident, they know about it immediately. This transparency has practical benefits: * **Reduced support load** - Users check the status page instead of contacting support * **Faster incident response** - Automated incident creation means faster communication * **Accurate SLA reporting** - Uptime calculations reflect real user experience [Learn how service uptime is calculated](/docs/communicate/status-pages/overview#service-uptime) with automated incidents. When your status page answers "can I use this?" instead of "are the servers up?", users pay attention. ## Further reading * [Status Pages Overview](/docs/communicate/status-pages/overview) - Complete reference for status page features * [Incident Management](/docs/communicate/status-pages/incidents) - Detailed guide to creating and managing incidents * [Subscriber Notifications](/docs/communicate/status-pages/subscriber-notifications) - Set up email notifications for status changes * [Anatomy of a Status Page](/docs/learn/incidents/anatomy-of-a-status-page) - What users expect from status pages # Automate Uptime Monitor Creation at Scale Source: https://www.checklyhq.com/docs/guides/create-multiple-monitors Monitoring hundreds or even thousands of pages doesn’t have to be a manual process. With Checkly’s Monitoring as Code approach, you can dynamically create, update, and manage URL monitors at scale. Checkly's model of [Monitoring as Code](https://www.checklyhq.com/guides/getting-started-with-monitoring-as-code/) lets you create site monitors right from your IDE. With the simple and scalable monitors of [Uptime Monitoring](https://www.checklyhq.com/blog/announcing-checkly-uptime-monitors-simple-scalable/), we have the ability to monitor every single URL, API route, and even TCP endpoint that our service provides. Since we're likely to want to monitor dozens or hundreds of URLs for complete coverage, this guide will show you how to create a large number of monitors all at once with a single file. ## How it Works ### Creating a Single URL Monitor With CLI Constructs Checkly's Monitoring as Code workflow starts with creating a [project](/docs/constructs/overview) where all your checks will be stored. You can use Checkly constructs to describe new Checkly monitors within your Checkly dashboard. If you haven't worked with Checkly at all, or have only created new monitors from our web UI, [install the Checkly CLI and create a new project for our checks with our getting started guide here](/docs/cli/overview). A basic URL monitor with minimal configuration looks like this: ```ts {title="first-url-monitor.check.ts"} theme={null} import { Frequency, UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs' new UrlMonitor('url-monitor-001', { // the monitor's unique logical ID name: 'Example URL Monitor', // display name to show in the Checkly UI activated: true, frequency: Frequency.EVERY_1M, maxResponseTime: 10000, // milliseconds request: { url: 'https://www.checklyhq.com/learn/', // target URL followRedirects: true, // allow 3xx responses skipSSL: false, assertions: [ //unlike more advanced checks, URL monitors only check the status code UrlAssertionBuilder.statusCode().equals(200), ] } }) ``` This uses the constructs API with components that are standardized between most check types. With the same constructs you can configure everything from the simplest URL monitor to an advanced browser check. A few details to note as we begin: * Each monitor needs a unique logical ID. By matching this ID we can update an existing monitor without deleting its history. * Except for the `request` object, logical ID, and name, every other configuration of a URL monitor can be defined either: * Individually on that single monitor * As part of a group of monitors * By using defaults set in the project configuration file (**`checkly.config.ts`**). If we save this file to our Checkly project's `/__checks__` folder, we can run this test or deploy it. Go ahead and save this check locally, and try replacing the URL provided with one from your own site, and then running the check with: ```bash theme={null} npx checkly test first-url-monitor ``` You should see feedback on your terminal as this check runs through the Checkly network. ```bash theme={null} __checks__/first-url-monitor.check.ts ✔ Example URL Monitor (287ms) 1 passed, 1 total ``` So far, so good - you could even run `npx checkly deploy` to deploy this to Checkly. But what we really want is to create dozens or hundreds of URL monitors all at once. We definitely don't want to create hundreds of individual files just to monitor every URL in our service (and we certainly don't want to manage all those files long-term 😅). Next, let's use this same file to create multiple monitors. ### Monitor Multiple URLs ```ts {title="urlArrayMonitors.check.ts"} theme={null} //In this bulk monitoring example we'll monitor several URLs from one file import { Frequency, UrlMonitor, UrlAssertionBuilder } from 'checkly/constructs' const sitemapUrls = [ 'https://www.checklyhq.com/docs/', 'https://www.checklyhq.com/learn/playwright/codegen/', 'https://www.checklyhq.com/docs/guides/setup-scripts-for-apis/', 'https://www.checklyhq.com/docs/api-reference/incidents/create-an-incident/', 'https://www.checklyhq.com/customers/' ] // Loop through each URL and create a monitor sitemapUrls.forEach((url, index) => { // Extract domain name for monitor ID const domain = new URL(url).hostname.replace('www.', '') // Generate unique logical ID for each monitor const logicalId = `multi-url-monitor-${index + domain}` new UrlMonitor(logicalId, { name: `URL Monitor - ${url}`, activated: true, frequency: Frequency.EVERY_5M, maxResponseTime: 10000, degradedResponseTime: 5000, request: { url: url, followRedirects: true, skipSSL: false, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) }) ``` The array we’ve provided in this check means that 5 separate URL monitors will be created if we deploy this code. Let’s try previewing a deployment with just `urlArrayMonitors.check.ts` in our `/__checks__` directory by running the command `npx checkly deploy -p`. ```bash theme={null} Parsing your project... ✅ Validating project resources... ✅ Bundling project resources... ✅ Create: UrlMonitor: multi-url-monitor-0checklyhq.com UrlMonitor: multi-url-monitor-1checklyhq.com UrlMonitor: multi-url-monitor-2checklyhq.com UrlMonitor: multi-url-monitor-3checklyhq.com UrlMonitor: multi-url-monitor-4checklyhq.com ``` *Each of our new monitors is listed in the deployment preview.* We've already saved a significant amount of time since we can simply add new URLs to our array to monitor additional paths. But, we can take this even further. ### **Monitoring URLs Stored in a Local File** In our previous check, we defined the URLs to monitor in an array at the top of the file: ```ts theme={null} const sitemapUrls = [ 'https://www.checklyhq.com/docs/', 'https://www.checklyhq.com/learn/playwright/codegen/', 'https://www.checklyhq.com/docs/guides/setup-scripts-for-apis/', 'https://www.checklyhq.com/docs/api-reference/incidents/create-an-incident/', 'https://www.checklyhq.com/customers/' ] ``` Rather than having to edit this check’s code every time we update our list of URLs, let’s make this more convenient for ourselves by reading a local JSON file with the same data. First, we’ll create `urls.json` ```json theme={null} { "urls":[ "https://www.checklyhq.com/docs/", "https://www.checklyhq.com/learn/playwright/codegen/", "https://www.checklyhq.com/docs/guides/setup-scripts-for-apis/", "https://www.checklyhq.com/docs/api-reference/incidents/create-an-incident/", "https://www.checklyhq.com/customers/" ]} ``` Then update our check to read the local file: ```ts theme={null} import * as urlsData from './urls.json' // Configuration: URLs to monitor const sitemapUrls = urlsData.urls ``` Now the monitored URLs are stored in a separate file that is easier for other engineers to read, access, and update. ### Monitor URLs Based by Fetching a CSV Let’s continue to increase the sophistication of our group of monitors. Right now our logical IDs are generated based on the URL. This leads to some names that aren’t very human-readable. Let’s have our check file look for a CSV file that has columns with all the data. Our CSV is formatted like this: | url | logicalID | name | | ---------------------------------------------------------------------------- | ----------- | ----------------------- | | `https://www.checklyhq.com/docs/` | docs-1 | url monitor docs-1 | | `https://www.checklyhq.com/learn/playwright/codegen/` | codegen-1 | url monitor codegen-1 | | `https://www.checklyhq.com/guides/using-pw-test-command/` | pw-test-1 | url monitor pw-test-1 | | `https://www.checklyhq.com/docs/api-reference/incidents/create-an-incident/` | incidents-1 | url monitor incidents-1 | And this version of the URL monitor fetches the CSV and parses the three values for each check from the row before creating a monitor. ```ts {title="csv-import-url-monitors.check.ts"} theme={null} import { Frequency, UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs' const csvUrl = 'https://raw.githubusercontent.com/serverless-mom/claudeExperiments/refs/heads/main/urlsWithIds.csv' interface CsvRow { url: string logicalId: string name: string } async function fetchUrlsFromCsv(csvUrl: string): Promise { try { const response = await fetch(csvUrl) if (!response.ok) { throw new Error(`Failed to fetch CSV: ${response.status} ${response.statusText}`) } const csvText = await response.text() return csvText .split('\n') .map(line => line.trim()) .filter(line => line.length > 0) .map(line => { //split each line and remove the quotation marks around the URLs const parts = line.split(',').map(part => part.trim().replace(/"/g, '')) return { url: parts[0], logicalId: parts[1], name: parts[2] } }) //remove the header row and any other invalid rows .filter(row => row.url.startsWith('http')) } catch (error) { console.error('Error fetching URLs from CSV:', error) return [] } } const urlsToMonitor = await fetchUrlsFromCsv(csvUrl) urlsToMonitor.forEach((row) => { new UrlMonitor(row.logicalId, { name: row.name, activated: true, frequency: Frequency.EVERY_5M, maxResponseTime: 10000, request: { url: row.url, followRedirects: true, skipSSL: false, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) }) ``` With these methods we could continue to extend the dynamic capabilities of this list of URL monitors, e.g., downloading and parsing a `sitemap.xml` for URLs to monitor. For the moment, though, let's move on to managing the configuration of multiple checks. ### Manage Configuration for Multiple Monitors with Groups As you handle monitors for multiple sites and environments, inevitably you'll want to manage the configuration for a number of monitors all at once. Currently, by creating many monitors from a single file we're giving each one the same configuration, and any configuration not specified will take the default values in `checkly.config.ts` in the project's root directory. We can create specific groups to receive configuration by adding the monitor to a [group with Checkly constructs](/docs/constructs/overview#checkgroupv2). Groups offer a convenient interface to configure your monitors, setting their retry strategy, alert thresholds, and where the group's alerts will be sent. Groups also [show up as folders in your Checkly web UI](/docs/platform/groups); making it easier to see the status of every check in the group. Checkly dashboard showing a check group with multiple monitors

Checkly dashboard showing a check group with multiple monitors

We can create a group with constructs: ```ts {title="prod-monitoring-group.ts"} theme={null} import { Frequency, CheckGroupV2, RetryStrategyBuilder } from 'checkly/constructs' export const prodMonitorGroup = new CheckGroupV2('prod-monitor-group', { name: 'Prod Monitor Group', activated: true, frequency: Frequency.EVERY_5M, locations: ['us-east-1', 'eu-west-1'], tags: ['url-monitoring', 'production'], environmentVariables: [], retryStrategy: RetryStrategyBuilder.fixedStrategy({ baseBackoffSeconds: 60, maxRetries: 2, maxDurationSeconds: 600, sameRegion: false }) }) ``` Now we can update the very first URL monitor we created to make it part of this group. ```ts theme={null} import { UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs' //import the group import { prodMonitorGroup } from './prod-monitoring-group.ts' new UrlMonitor('url-monitor-example', { name: 'Example URL Monitor', group: prodMonitorGroup, //add this check to the group //most config can now be handled at the group level; //response time thresholds are always set per monitor maxResponseTime: 10000, request: { url: 'https://httpbin.org/get', followRedirects: false, skipSSL: false, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) ``` This single monitor’s file is now quite a bit more compact and clean, with more of the configuration handled at the group level. ## Conclusion: Scale Your Monitoring Effortlessly with Checkly Monitoring hundreds or even thousands of pages doesn’t have to be a tedious, manual process. With Checkly’s **Monitoring as Code** approach, you can dynamically create, update, and manage URL monitors at scale—whether from a local JSON file, a CSV, or even a sitemap. Remember that this approach works for more than just URL monitors, you can extend the automatic creation of monitors to [API checks](/docs/detect/synthetic-monitoring/api-checks/overview) and even [browser checks](/docs/detect/synthetic-monitoring/browser-checks/overview). ### Key Takeaways: * **Bulk Monitoring Made Simple**: Define multiple URL monitors in a single file using arrays, loops, or external data sources like CSVs. * **Centralized Configuration**: Use **Groups** to standardize settings like frequency, retries, and alerting across multiple monitors. * **Dynamic Updates**: Modify your monitoring setup without losing historical data by leveraging unique logical IDs. ### Next Steps * **Explore Advanced Checks**: Beyond URL monitoring, Checkly supports [API checks](/docs/detect/synthetic-monitoring/api-checks/overview) with deeper validation and setup/teardown scripts, and [browser checks](/docs/detect/synthetic-monitoring/browser-checks/overview) for full user flow validation. * **Set Up Alerts**: Configure [alert channels](/docs/communicate/alerts/overview) to notify your team of issues via Slack, email, or PagerDuty. * **Automate Deployments**: [Integrate Checkly into your CI/CD pipeline](/docs/integrations/ci-cd/overview) to ensure monitoring updates with every code change. Ready to scale your monitoring? [Sign up for Checkly](https://www.checklyhq.com/) today and deploy your first batch of monitors in minutes! **Further Reading:** * [Checkly CLI Documentation](/docs/cli/overview) * [Monitoring as Code Best Practices](https://www.checklyhq.com/guides/monitoring-as-code-best-practices/) * [The Defense-in-Depth Approach To Application Monitoring](https://www.checklyhq.com/blog/the-defense-in-depth-approach-to-app-monitoring/) Have questions? Join the [Checkly Community Slack](https://checklyhq.com/slack) to chat with other developers and the Checkly team. Happy monitoring! 🚀 # Automating Check Steps for Developers Source: https://www.checklyhq.com/docs/guides/developer-fixtures When we enable developers to create page or API monitors, we want to automate as much of the code writing process as possible, so that developers can test the functionality they care about without getting bogged down in setup steps. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. ## To involve developers in monitoring, you need to make it easy Most engineering organizations are shifting their monitoring and observability left, making developers part of the team that makes sure their service is always running and available. But we don't expect our developers to become QA and Operations experts, so we need to enable their contributions with as much automation and default configuration as possible. The goal of this guide is to help create an easy process for all your engineers to contribute to monitoring. Just like any Checkly user, developers can still control every aspect of how a monitor runs, when it detects failure, and who gets notified of that failure. This isn't about taking away control, just setting defaults and automation so that you don't need to become a monitoring expert to contribute to your team's monitoring. This guide will tie together material from our Playwright Learn articles, our blog, and our tutorial videos, to help you create an environment for developers that makes it easy for them to add new page checks with minimal repetitive code. We'll cover: * The Checkly CLI and Monitoring as Code for developers * Creating standardized configuration for checks and groups of checks * Automating repeated tasks like account logins so the code can be re-used in multiple checks * Adding standardized code that runs with every check ### Whom this guide can help This article will refer repeatedly to an ideal "developer" who understands their own service quite well, but isn't experienced at either QA or monitoring. They may not be an expert in the automation framework Playwright, which we use to write our monitors, and they don't have experience with specific configuration like retry logic, alert thresholds, etcetera. This guide can also help you make monitoring accessible for: * Front-end developers who work on the user experience, but haven't previously monitored the site in Production * Integration engineers who want to make sure that changes to third-party services don't break your site * User Experience engineers who want to make sure everything works as expected for all users Though the rest of this guide will refer to our developer as the person we're enabling, feel free to slot in anyone who we want to enable to write monitors, without requiring they become experts in Observability, Monitoring, Checkly, or Playwright. ## The Checkly CLI and Monitoring as Code for developers First off, our developer may not be very excited to log into a web interface to create new monitors. Since developers are used to running and deploying code from the command line, we want to make available the Checkly CLI to our developers. A [full guide to the Checkly CLI](https://www.checklyhq.com/cli/overview) is on our documentation site, but in general the process would look like: 1. Set up the CLI on the developer's machine - install is easy with npm, then the developer can authenticate their cli with `npx checkly login`. 2. Create a new project - developers can either grab an existing repository with the team's checks, or use `npx checkly init` to make a new project. 3. Run your checks - rather than running new Playwright checks from the developer's laptop, they can run checks through the whole Checkly system with `npx checkly test`, the tool will scan the project for tests, run them as configured from the real geographic locations, and give a local report on results. 4. Deploy your checks - Once the developer is happy with their checks, they can deploy them with `npx checkly deploy`, and they'll show up for all users in the web interface. For small teams just getting started with Checkly, this is all you need to do to harness the CLI for your process, but as you grow you'll want to have a single source of truth for the code of all existing checks, and if necessary a change review process, for that you can use the [Monitoring as Code](https://www.checklyhq.com/guides/monitoring-as-code/) model by storing your checks and their configuration in a shared repository. ## Grouping Checks and Standardizing Configuration It's a great feature of Checkly that any check can have totally custom settings for every aspect of its running: it can run at a custom cadence, from its own geography, with its own logic for retries. This means that if, for example, you're creating a new test specifically to check localization settings, you can set just that check to run from dozens of geographies, even if most checks only need three or four. Of course, we don't want to give our fresh developer over a dozen settings that she needs to set just to create her first check. New Checks created either in the web UI or from your code environment via the CLI will default to using the [global configuration](/docs/constructs/overview#global-configuration). You can view and edit this global config in `checkly.config.ts`, you can see a reference on this configuration and options under the `project` [construct documentation](/docs/constructs/project). Some suggestions for default configuration: * Set a generous `RetryStrategy`! Some failures can look extremely worrying even if they happen once, but it's worth double-checking that the problem wasn't entirely ephemeral. Full [documentation of retry configuration is on our docs site](/docs/constructs/retry-strategy). * If you're encouraging microservice developers or frontend engineers to add checks, consider setting the default frequency to be relatively low. It's great to test edge cases or unusual scenarios, but someone without exposure to Operations is unlikely to need to run a check every minute. * Make sure your geographic `locations` reflect your userbase. You can use `runParallel:false` if testing from a large number of locations, so that checks will run from a single location each cycle. ### Check Groups and Alert Policies You may want to go beyond global settings for a larger or more complex team. For example if a group of engineers is working with: * Different notification expectations * Different URL's and other check variables * A specific set of features in your application These are all goud reasons to create a check group. Groups have their own set of variables and configuration for all the members of that group. Checks can be added to a group either by configuration at the code level (passing a `checkGroup` object to the config), or in the web UI by creating a group and clicking 'Add Checks to Group'. the Checkly Web UI with settings for a group, displaying variables

the Checkly Web UI with settings for a group, displaying variables

Some important things to remember when working with a group of checks: * Alerts go to the groups alert channels, so be sure your group is connected to at least one alert channel or you will miss any notifications from this group! * Checks in the group are run individually, not as a group. If you're thinking of doing something like a post-run cleanup with a single check for the whole group, that won't work as expected, instead run cleanup scripts individually for each check. * For API checks, you can have a 'teardown script' for a group that runs after every check is completed. * That said, all the checks in a group can be triggered to run at the same time, this is especially useful if a group is used to run checks after deployments. the Checkly Web UI with settings for a group

the Checkly Web UI with settings for a group

* Any settings that are given in two places will be reconciled with the following hierarchy: 1. individual settings on a check 2. group settings 3. global settings So group settings will override global settings, and individual settings will override everything else. * While groups are a useful feature of Checkly, and will make sense for some applications, you may also be served by using Javascript and Typescript's native features for [inheritance](https://www.typescriptlang.org/docs/handbook/2/classes.html). Since files can inherit modules from others, it's possible to share configuration very cleanly, is as complex a heirarchy as you'd like, using these native features. This especially may make sense if you're going deep into [Monitoring as Code](/docs/guides/getting-started-with-monitoring-as-code) and want to set up a complex repository for all monitoring. Native features also help support the next section on using fixtures to share code blocks. ## Sharing code for repeated tasks While there are sophisticated ways to share authentication across checks discussed in other guides, it's good to know how to share code with fixtures for common repeated tasks. Let's imagine an example where we've got a web shop with a login modal that we need to fill out before doing further account actions. The goal of my check isn't really to test this modal, it just needs to be filled out before my *real* testing, where I look at my demo user's recent orders. To open a modal and log in, we might use some lines like the following: ```ts RecentOrders.test.ts theme={null} import { test, expect } from '@playwright/test'; test('display Recent Orders', async ({ page }) => { await page.goto('https://danube-webshop.herokuapp.com/'); await page.getByRole('button', { name: 'Log in' }).click(); await page.getByPlaceholder('Email').click(); await page.getByPlaceholder('Email').fill('robert@moresby.com'); await page.getByPlaceholder('Password').click(); await page.getByPlaceholder('Password').fill('@!#KLjwsWe4@'); await page.getByRole('button', { name: 'Sign In' }).click(); // Here's my actual test code to check my orders }); ``` This is a fine practice for a single test, but if we have dozens that all require login as a first step, we'll find ourself copying and pasting this code several times. This means a lot of copied code, such that if we ever tweak the login process, we'll have to update this copied code in dozens of places, not good! We'd want to move this code into an extension of the Playwright page fixture like so: ```ts RecentOrdersWithFixture.test.ts theme={null} import { test as baseTest, expect } from '@playwright/test'; const test = baseTest.extend({ userPages: async ({ page }, use) => { // Your setup logic here await page.goto('https://danube-webshop.herokuapp.com/'); await page.getByRole('button', { name: 'Log in' }).click(); await page.getByPlaceholder('Email').click(); await page.getByPlaceholder('Email').fill('robert@moresby.com'); await page.getByPlaceholder('Password').click(); await page.getByPlaceholder('Password').fill('@!#KLjwsWe4@'); await page.getByRole('button', { name: 'Sign In' }).click(); console.log('setup'); // After logging in, pass the `page` // object to the test cases // // Whatever you call `use` with // will be available in your test cases await use(page); // Teardown logic, if any console.log('teardown'); }, }); //Now we can use that userPages object repeatedly test('display recent orders', async ({ userPages }) => { // any code added here can make normal expectaions of the page! }); test('cancel recent order', async ({ userPages }) => { // any code added here can make normal expectaions of the page! }); ``` In this example, we've just put our fixtures at the top of a single check's file, and shared it across tests within that file, in the next we'll share a code snippet across multiple files, and running code with every check. ### See it in action: Reuse Playwright Code across Files and Tests with Fixtures To see fixtures demonstrated, and a step-by-step explanation of the code, check out [Stefan's tutorial video](https://www.youtube.com/watch?v=2O7dyz6XO2s\&list=PLMZDRUOi3a8N067UNvkxXEThKlTII_OJ-\&index=5): ## Running standardized code across every check While it's useful to package up tasks like logging in across multiple tests, we may want some code to run before and after every single check. Some possible examples: * We know that every session has to set the same page settings every time (like cookie settings) * Manually cleaning up any test's actions - for example restoring our demo user's account settings For these, again, we could use core Javascript/Typescript modules to accomplish this task, here's an example ```ts EcommerceActions.test.ts theme={null} import {setup, teardown } from "./standardScripts.ts" test.beforeEach(() => { setup(); }); //add tests here test.afterEach(() => { teardown(); }); ``` But while it's great to use native features, this is requiring a good amount of setup in each test file (the above snippets would have to be repeated in each new file), and our goal is to make creating new checks as easy as possible for developers. So let's use Playwright's native features to make this even easier for them. ```ts TestFixtures.ts theme={null} import { test as base } from '@playwright/test'; export const test = base.extend<{ forEachTest: void }>({ forEachTest: [async ({ page }, use) => { // This code runs before every test. console.log('test is about to start! setting up'); await use(); // This code runs after every test. console.log('test finished! cleaning up'); }, { auto: true }], // automatically starts for every test. }); ``` Now the only change you'll need to make to your test files is to have them import the `test` function from this fixture file, for example: ```ts EcommerceActions.test.ts theme={null} import { test } from './testFixtures'; import { expect } from '@playwright/test'; ``` ### See it in action: running code before and after every check To see this demonstrated, and a step-by-step explanation of the fixture code, check out [Stefan's tutorial video](https://www.youtube.com/watch?v=hegZS46J0rA\&list=PLMZDRUOi3a8N067UNvkxXEThKlTII_OJ-\&index=1): Here's our [complete YouTube playlist on fixtures](https://www.youtube.com/playlist?list=PLMZDRUOi3a8N067UNvkxXEThKlTII_OJ-). ## Conclusions Simplifying monitoring processes through automation and standardized practices allows developers to focus on what matters: ensuring their services function as intended. By leveraging tools like the Checkly CLI and adopting practices like Monitoring as Code, reusable fixtures, and standardized configurations, teams can involve developers in monitoring without overwhelming them with setup complexity or configuration details. These strategies reduce repetitive work, enable consistency across tests, and ensure that monitoring integrates seamlessly into the development workflow. Ultimately, this approach promotes a culture of shared responsibility for reliability, making it easier for developers to contribute to the stability and observability of their systems without needing deep expertise in monitoring tools. With these methods, teams can improve collaboration and maintain a robust monitoring setup that scales with their needs. # Empowering DevOps with Monitoring as Code Source: https://www.checklyhq.com/docs/guides/empowering-developers-with-checkly Discover how Checkly empowers developers and platform teams to streamline complex monitoring through a code-first approach. Learn how collaboration, automation, and integrated alerts improve reliability and reduce bottlenecks in modern software delivery workflows. ## Why developers need to contribute to monitoring In today’s fast-paced development environment, engineers are under pressure to ensure that their applications not only function but also perform reliably under a wide variety of circumstances and conditions. Effective monitoring is key to staying on top of potential issues and keeping systems running smoothly with minimal downtime. However, in many organizations, the process of setting up and managing monitors falls to specialized platform or operations teams, which can create bottlenecks. Checkly solves this problem by providing a solution that bridges the gap between application developers and platform teams. By leveraging Checkly’s codified MaC approach, both groups can collaborate efficiently to create, configure, and manage monitors in a seamless way that fits within existing workflows. ## Empowering Developers with Code-First Monitoring With the rise of shift-left and the age of empowering engineers, more teams are using code to configure tests, infrastructure, and deployment models. They are finding benefits like increased collaboration, auditability, and automation in these new paradigms that are revolutionizing the way they ship software. Checkly fits neatly into this trend, offering software teams a codified approach to building and configuring their monitors and alerts. This means that monitors can be: * Created faster within the software delivery lifecycle * Tested and reviewed in CI/CD pipelines * Automated across services and teams No more throwing monitoring over the proverbial wall. Rather than relying on a separate platform or operations team to set up monitors, engineers can take full control of what gets monitored, when, and how. This can save time, reduce errors, and make the entire process more efficient. For example, here’s a simple API check defined in Checkly’s CLI: ```bash Terminal theme={null} curl --request POST \ --url 'https://api.checklyhq.com/v1/checks/api?autoAssignAlerts=true' \ --header 'Authorization: [your-key]' \ --header 'accept: application/json' \ --header 'content-type: application/json' \ --data ' { "activated": true, "muted": false, "shouldFail": false, "alertSettings": { "escalationType": "RUN_BASED", "reminders": { "amount": 0, "interval": 5 }, "runBasedEscalation": { "failedRunThreshold": 1 }, "parallelRunFailureThreshold": { "enabled": false, "percentage": 10 } }, "useGlobalAlertSettings": true, "groupId": null, "groupOrder": null, "runtimeId": null, "retryStrategy": { "type": "FIXED", "baseBackoffSeconds": 60, "maxRetries": 2, "maxDurationSeconds": 600, "sameRegion": true }, "runParallel": false, "request": { "method": "GET", "url": "https://api.checklyhq.com", "skipSSL": false, "ipFamily": "IPv4", "bodyType": "NONE" }, "frequency": 1, "tearDownSnippetId": null, "setupSnippetId": null, "localSetupScript": null, "localTearDownScript": null, "degradedResponseTime": 10000, "maxResponseTime": 20000, "name": "webshop-api", "locations": [ "us-east-1", "ap-east-1" ] } ' ``` As seen here, engineers can quickly configure an API check that runs every minute to ensure that the status code is 200. If there’s a failure, Checkly will immediately notify the team, allowing them to address the issue promptly. ## Collaboration Between Dev and Platform Teams While a code-first approach to monitoring empowers application engineers, many teams include both developers and platform engineers who work together to build and operate complex systems. This is where Checkly’s flexibility and extensibility truly shines. Platform teams often handle the configuration of complex alerts, thresholds, and scheduling across multiple environments. By codifying these aspects, platform engineers can provide a consistent monitoring “wrapper” around the application teams’ checks. This allows developers to focus on building and shipping code and adding simple checks without worrying about the operational intricacies of monitoring. For example, here’s a more complex Checkly configuration, where the platform team might define specific response time thresholds and alerting rules: ```bash Terminal theme={null} resource "checkly_check" "example_check_2" { name = "Example API check 2" type = "API" activated = true should_fail = true frequency = 1 degraded_response_time = 5000 max_response_time = 10000 locations = [ "us-west-1", "ap-northeast-1", "ap-south-1", ] alert_settings { escalation_type = "RUN_BASED" run_based_escalation { failed_run_threshold = 1 } reminders { amount = 1 } } retry_strategy { type = "FIXED" base_backoff_seconds = 60 max_duration_seconds = 600 max_retries = 3 same_region = false } } ``` In this example, the platform team has set up detailed monitoring parameters, including response time thresholds and a retry strategy in case of failure. By wrapping these details into reusable configurations, the platform team allows application engineers to create new monitors that are consistent with the organization’s standards—without having to worry about the operational details. ### Codified Alerts and Notifications Checkly also integrates alert channels into the code, allowing teams to manage alerts for different monitors via a code-first approach. You can specify email alerts, Slack notifications, or other channels to ensure that the right team members are notified when something goes wrong. For instance, here’s how to set up email alerts for a check: ```bash Terminal theme={null} resource "checkly_alert_channel" "email_ac1" { email { address = "info1@example.com" } } resource "checkly_check" "example_check" { name = "Example check" alert_channel_subscription { channel_id = checkly_alert_channel.email_ac1.id activated = true } } ``` By codifying alert configurations, platform engineers can ensure that the organization’s monitoring rules and notification protocols are followed consistently, even as application teams create new monitors. ## Conclusion Checkly’s approach to monitoring via code gives engineering and operations teams the tools they need to keep applications running smoothly. Application engineers can take ownership of their monitors, ensuring that they’re set up efficiently and integrated into their workflows. Meanwhile, platform teams can manage and maintain a higher-level view, providing the necessary configurations and support for more complex systems. Whether your team is fully developer-led, or you have a more traditional split between development and platform engineering, Checkly’s code-first monitoring solution ensures that everyone can collaborate smoothly and efficiently. As modern applications continue to grow in complexity, tools like Checkly are becoming essential to manage the intricacies of monitoring in a fast-moving development environment. # What is End to End Monitoring? An Overview with Examples Source: https://www.checklyhq.com/docs/guides/end-to-end-monitoring Learn end-to-end monitoring with playwright to test key website flows. Follow our guide that gets you up and running in 10 minutes. ## An overview of end-to-end monitoring End-to-end monitoring is a comprehensive strategy designed to assess the overall health and performance of a service by examining the entire lifecycle of requests to an application. By simulating a user request, including the browser rendering and user experience, an operations or development professional tests all parts of an application at once; hence the name ‘end-to-end.’ End-to-end monitoring is an outgrowth of end-to-end testing, where rather than sending test request once, with monitoring tests are run on a cadence, and the results automatically compiled. By aligning monitoring insights with the user experience, end-to-end monitoring ensures that any anomalies, edge cases, or potential failures impacting users are quickly identified and addressed. This inclusive monitoring strategy often involves multiple teams, including developers, network engineers, and non-technical stakeholders, fostering a collaborative effort to maintain optimal system performance and enhance the digital experience for users. ## End-to-end monitoring vs traditional monitoring approaches Unlike traditional testing, which focuses on isolated components and only tests occasionally, end-to-end monitoring provides a holistic view of a product or service. Methods like unit testing look at individual aspects like servers or databases, sending specific requests to the services and verifying the response. This kind of testing doesn’t allow for edge cases like display issues, unusual accounts with different settings, or other problems that real users experience after unit testing. Running tests only occasional, the traditional approach of end-to-end testing only reveals predictable failures, e.g. when major code changes break the service. By running tests on a cadence, more unexpected failures are revealed by end-to-end monitoring. ## Headless browser testing Over the course of the last decade, especially thanks to tools such as [Selenium](https://www.selenium.dev/) and (more recently) [Cypress](https://www.cypress.io/), **automated End-to-End testing (E2E testing) has become widespread across industries**. Broadly speaking, **E2E testing entails running fully automated test suites with the goal of catching bugs before they hit production** and, therefore, negatively affect the user experience. These test suites need to be carefully scripted using dedicated tools, as well as to be made stable and fast enough to test the most important end-user flows on every build, PR or commit, depending on the application under test and the organisation's automation maturity. The industry has learned to struggle with the challenges this approach presents: 1. Long-running suites. 2. Test flakiness. 3. Expensive test infrastructure. **All of the above lead to higher costs and slower delivery.** The appearance of mature **headless browser automation tools, such as [Playwright](https://playwright.dev), offer a response** to many of the above issues by allowing testing in the browser without its GUI, which yields higher speed and stability coupled with lower resource consumption. ## E2E monitoring examples While this nimbler, more reliable kind of test is already a big improvement for pre-production testing, it enables a completely new approach in production monitoring: we can now **continuously run E2E tests against our production systems**. This enables us to have real-time feedback on the status of our website's key user flows from a user's perspective. This is E2E monitoring, also known as *synthetic monitoring* or *active monitoring*. This comes with a significant, often underestimated advantage: it allows us to **catch all those things that might break in production that can't be caught during pre-production testing**. We are now running directly against the system that the end-user is actually interacting with, and will be able to monitor its behaviour in real time. What could this look like in practice? Let's look at an e-commerce example. ## Monitoring a web shop A few key flows for an e-commerce websites could be: 1. Logging in 2. Finding a product through search 3. Adding products to the basket and checking out Let's see how to set them up - for this example, we will do that on our [demo web shop](https://danube-web.shop/). demo website screenshot

### Playwright E2E tests Using Playwright, we can script our three E2E scenarios as follows: ```js login.spec.js theme={null} import { test, expect } from '@playwright/test'; test('web shop login', async ({ page }) => { //navigate to our web page await page.goto('https://danube-web.shop/'); await page.getByRole('button', { name: 'Log in' }).click(); //for your own application, you may want to encode login details in a config file await page.getByPlaceholder('Email').click(); await page.getByPlaceholder('Email').fill('user@email.com'); await page.getByPlaceholder('Password').click(); await page.getByPlaceholder('Password').fill('supersecure1'); await page.getByRole('button', { name: 'Sign In' }).click(); //check for a confirmation message on the page await expect(page.getByText('Welcome back, user@email.com')).toBeVisible(); }); ``` ```js search.spec.js theme={null} import { test, expect } from '@playwright/test'; [ { title: 'The Foreigner', author: 'Camuso' }, { title: 'Does the Sun Also Rise?', author: 'Doubtingway' }, { title: 'The Transformation', author: 'Fafka' }, ].forEach(({ title, author }) => { // You can also do it with test.describe() or with multiple tests as long the test name is unique. test(`Searching For ${title}`, async ({ page }) => { await page.goto('https://danube-web.shop/'); await page.getByRole('textbox').click(); await page.getByRole('textbox').fill(title); await page.getByRole('button', { name: 'Search' }).click(); await expect(page.locator('#app-content')).toContainText(author); }); }); ``` ```js checkout.spec.js theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { //navigate to page await page.goto('https://danube-web.shop/'); //select item await page.getByText('Haben oder haben').click(); await page.getByRole('button', { name: 'Add to cart' }).click(); await page.getByRole('button', { name: 'Checkout' }).click(); //fill out checkout info await page.getByPlaceholder('Name', { exact: true }).fill('nocnica'); await page.getByPlaceholder('Surname').click(); await page.getByPlaceholder('Surname').fill('mellifera'); await page.getByPlaceholder('Address').click(); await page.getByPlaceholder('Address').fill('123 fake street'); await page.getByPlaceholder('Zipcode').click(); await page.getByPlaceholder('Zipcode').fill('02183'); await page.getByPlaceholder('City').click(); await page.getByPlaceholder('City').fill('Springfield'); await page.getByPlaceholder('Company (optional)').click(); await page.getByPlaceholder('Company (optional)').fill('checkly'); //click buy await page.getByRole('button', { name: 'Buy' }).click(); await page.getByLabel('as soon as possible').check(); await expect(page.getByText('All good, order is on the way')).toBeVisible(); }); ``` These can be run on our own machine without issues with [very little preparation](https://playwright.dev/docs/intro) with a simple `node script.js`. ## Monitoring application performance A web application's performance plays a primary role in the user experience it delivers. From the user's perspective, a fully functional application that is not performant quickly becomes indistinguishable from a broken one. Using Playwright together with browser APIs or additional performance libraries, our end-to-end monitoring setup can be easily extended to include application performance. ## End to end application monitoring Unlike headful tools, headless ones tend to not be very resource-hungry, which makes it easier to move our scripts to the cloud. Checkly runs on top of AWS Lambda, and enables us to quickly copy-paste our script and set it up to run on a schedule from locations around the world. checkly check creation screenshot

We can move our scripts to separate checks to keep them [independent](/docs/learn/playwright/writing-tests#keep-tests-independent) - we want to optimise for parallelisation and clarity of feedback. checkly dashboard screenshot

As soon as a check runs red, we are alerted in real time and can **intervene before the issue impacts our users**. Alerting can be set up with all the industry-standard channels like Pagerduty, Opsgenie, Slack, email, SMS and more. checkly alerting screenshot

### On-demand checking **Active monitoring and event-triggered testing do not exclude one another.** You might want to have checks kicked off every time you deploy to production, or on every merge, PR or commit, or you might also want to run against your staging or development server. The choice has to be made based on your workflow and on your automation strategy. ## CI/CD Tests can be kicked off of CI pipelines. You might want to use different hooks (for e.g. smoke vs regression testing) in different stages and against different targets. [Checkly supports all major CI servers](/docs/integrations/ci-cd/overview). ## Develop-preview-test If you are using provider like Vercel, you can automatically trigger your checks to run on deployed PRs, too, to reap the benefits of the [develop-preview-test approach](https://rauchg.com/2020/develop-preview-test). ## Benefits of implementing end-to-end monitoring Now, let’s see what benefits you will experience if you decide to implement end-to-end monitoring. ### Improved user experience End-to-end monitoring identifies issues before they impact users, ensuring smooth and reliable interactions. This proactive detection boosts customer satisfaction and loyalty. Enhanced user experience directly translates to increased engagement and retention rates. ### Better operational efficiency By offering a complete system view, end-to-end monitoring quickly pinpoints and resolves bottlenecks. This reduces downtime and enhances productivity. Operational efficiency improvements lead to cost savings and better resource allocation. ### Faster issue resolution End-to-end monitoring means continuous monitoring, which enables rapid identification of root causes, reducing the time needed to address problems. This leads to quicker resolutions and minimizes disruptions. Faster issue resolution ensures continuous service availability and reliability. ### Actionable data With data from various sources, end-to-end monitoring provides a holistic understanding of system performance. This broad perspective helps in making informed decisions and optimizing resources. Comprehensive insight allows for better planning and strategic improvements. ### Collaboration across teams End-to-end monitoring fosters collaboration among developers, network engineers, and non-technical staff. This unified approach ensures all stakeholders work together to maintain system health and improve performance. Collaborative efforts lead to more innovative solutions and streamlined processes. ## Tools for end-to-end monitoring There are countless tools out there that perform some type of end-to-end monitoring. Here, we’re covering some of them: ### Application Performance Monitoring (APM) Tools *Tools:* New Relic, Dynatrace, AppDynamics APM focuses on tracking the performance and availability of software applications. It monitors key metrics such as response times, transaction throughput, error rates, and user satisfaction. APM helps identify performance bottlenecks and ensures applications run smoothly, providing a better user experience. ### Infrastructure Monitoring Tools *Tools:* Zabbix, Datadog, Prometheus Infrastructure monitoring involves overseeing the physical and virtual components of an IT environment, such as servers, storage systems, and virtualization platforms. It checks the status, performance, and resource utilization of these components to ensure optimal operation and prevent downtime. ### Real User Monitoring (RUM) *Tools:* New Relic Browser, Dynatrace RUM RUM tracks the actual interactions of users with a website or application in real-time. It captures data on load times, errors, and user behavior to provide insights into the real-world performance of digital assets. RUM helps identify areas for improvement based on actual user experiences. Since RUM is capturing data on every user every time, the infrastructure cost is nontrivial and there's always the risk of impacting performance for users ### Synthetic Monitoring Tools *Tools:* Checkly, Pingdom, Datadog Synthetic monitoring uses scripted tests to simulate user interactions with applications and services. It helps identify performance issues and availability problems before they impact real users. This proactive approach ensures continuous service reliability and performance. ## 4 Best practices for effective end-to-end monitoring ### Establishing clear objectives and KPIs Define specific goals and key performance indicators (KPIs) to measure the success of your monitoring strategy. This ensures that monitoring efforts are aligned with business objectives and provides a clear framework for evaluating performance. ### Collaborative approach: developers, security, operations, and business teams Foster collaboration among developers, security, operations, and business teams to ensure comprehensive monitoring coverage. This multidisciplinary approach helps identify and address issues from multiple perspectives, improving overall system health. ### Continuous monitoring and regular performance reviews Implement continuous monitoring to detect and address issues in real time. Regularly review performance data to identify trends, optimize resources, and make informed decisions for future improvements. ### Integrating end-to-end monitoring into DevOps and agile workflows Embed end-to-end monitoring into DevOps and agile workflows to ensure seamless integration and real-time feedback. This approach promotes a proactive monitoring culture and accelerates the identification and resolution of issues. The best way to do this is to adopt Monitoring as Code (MaC), and implement your monitors directly in your CI/CD, from pre-prod to prod. Checkly’s CLI enables you to build and maintain your monitors from your existing code base, integrating with cutting-edge testing and monitoring tools like Playwright, Grafana, Prometheus, Coralogix, and others. ## End-to-end Monitoring Pitfalls We learned things the hard way so you do not have to. When starting out, keep an eye out for the following pitfalls: * Non-independent tests: tests that rely on one another in any way (e.g. order of execution, test data) are hard to parallelise, resulting in longer execution times and potentially higher flakiness. [Keep your tests independent](/docs/learn/playwright/writing-tests#keep-tests-independent). * Long, unfocused tests: checking too much in a single test will make failures harder to debug. [Break it up instead](/docs/learn/playwright/writing-tests#keep-tests-focused), and enjoy the added parallelisation. * Messing up your own metrics, KPIs: remember, if you are not running against production, you want to make sure your E2E monitoring checks or tests are filtered out of your analytics. This is [rather easy to do](/docs/platform/allowlisting-traffic), with most headless browser tools normally identifying themselves as such from the start. ## How to deploy end-to-end monitoring with Checkly Now, let’s put the stuff we talked about in action. Deploying end-to-end monitoring with Checkly is super simple: Step 1: Setting Up a Checkly Account Ensure you have an active Checkly account. If not, sign up. The default free Team trial lasts for 14 days. To access advanced features beyond the trial, subscribe to a pricing plan based on your needs. Step 2: Setting Up Checkly in Repository After completing the installation steps, open a terminal in the directory of your project and run the following command: `npx checkly init` This command will bootstrap your repository with the basics needed to start using Checkly MaC in your project. Checkly CLI project setup screenshot

In your project directory, you will find a folder named “**checks**” containing the following check templates: ``` |__checks__ |- api.check.ts |- heartbeat.check.ts |- homepage.spec.ts ``` Once this setup is complete, log in to your Checkly account via the CLI using the following command: `npx checkly login` You can choose to log in from the browser or in your terminal. After logging in, you'll be able to update Checkly Checks from your local machine as long as you're connected to the internet. Step 3: Writing Your Monitoring Scripts In your development environment, write JavaScript/TypeScript tests for your code updates, similar to unit tests. We typically use the Playwright testing framework in the `.spec.ts` or `.check.ts` file. Consider a scenario where you want to monitor the title of the Checkly documentation and take a screenshot of the page. To do this, replace the code in the `homepage.spec.ts` with the following: ```javascript homepage.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('Checkly Docs', async ({ page }) => { const response = await page.goto('https://www.checklyhq.com/docs/browser-checks/'); // Ensure the page is loaded successfully expect(response?.status()).toBeLessThan(400); // Check if the page title is as expected const pageTitle = await page.title(); const expectedTitle = 'Introduction to Checkly | Checkly'; expect(pageTitle).toBe(expectedTitle); // Optionally, you can take a screenshot if needed await page.screenshot({ path: 'homepage.jpg' }); }); ``` This test uses the `page.goto` method to navigate to the specified URL (`https://www.checklyhq.com/docs/browser-checks/`). The method returns a response object, which is stored in the response variable. Then we use the `expect` function to assert that the HTTP status code of the response is less than 400. This is a way to ensure that the page is loaded successfully without any HTTP errors. `page.title()` retrieves the title of the page and compares it with the expected title ('Introduction to Checkly | Checkly') using the `expect` function. This ensures that the page title matches the expected value. Finally, we take a screenshot of the page and save it as 'homepage.jpg'. Step 4: Running Test Sessions Now that we have our test scripts ready, let's execute them. Run the following command in the terminal of your project repository to test your checks against Checkly's infrastructure: `npx checkly test` You can review the test session with full logging, videos and traces in the Checkly web interface. Here is the result of the test we just executed: Running tests from the CLI

There are also links to the detailed summary of the test at the end of the result in the terminal. Here is an example of the test summary: A test summary in the Checkly web interface

A test summary in the Checkly web interface

As seen in the result, the test failed because if you browse the URL (`https://www.checklyhq.com/docs/browser-checks/`) the title of the site is “Getting started | Checkly” and not “Introduction to Checkly | Checkly” as expected in the test case. If we update the test case to expect “Getting started | Checkly” we will have a passed test. Here is the result of the test after updating the correct title: A passing test

If you check the detailed summary, we should have a passed test too: A passing test

Step 5: Deploying Checks Now that you've reviewed and updated your tests, you can proceed to deploy your MaC workflow and related resources, such as alerts and dashboards. Run the following command in your project terminal to deploy the tests to your Checkly account: `npx checkly deploy` Once the deployment is complete, you'll see a success message in your terminal, indicating that the project has been deployed to your Checkly account. To verify this, navigate to the home section on the left side of the Checkly UI, and you'll find the project with the name of the test script from your local repository. A passing test

Step 6: Setting up Alerts Checkly offers alert services to notify you whenever a check fails. Various alert channels are available, including Slack, SMS, webhook, phone call, email, Opsgenie, PagerDuty, etc. To set up alerts for your check, go to the specific project, in this case, "homepage.spec.ts." At the top right corner of the code editor, click the "Settings" button. In the revealed side panel, access "Alert Settings" under "Retries & alerting." A passing test

Why should the way we manage monitoring be any different? Understand monitoring as code (MaC) via our Checkly CLI. Setup scripts are a fundamental tool to tailor API checks to your own target endpoints.

# Getting Started with Monitoring as Code Source: https://www.checklyhq.com/docs/guides/getting-started-with-monitoring-as-code Monitoring is essential to ensure that your applications run smoothly, efficiently, and reliably. As systems grow more complex, the need for scalable and automated monitoring has never been greater. This is where Monitoring as Code (MaC) comes into play, allowing you to define, deploy, and manage monitoring configurations programmatically. Checkly offers a powerful Command Line Interface (CLI) that enables teams to implement Monitoring as Code seamlessly. This article will guide you through the initial steps of getting started with Checkly's CLI to set up your MaC environment. ## Why Monitoring as Code? Before diving into the specifics of Checkly’s CLI, it's important to understand why MaC is beneficial: 1. **Version Control**: With MaC, you can store your monitoring configurations in a version control system, such as Git. This allows you to track changes, revert to previous configurations, and collaborate with your team more effectively. 2. **Scalability**: MaC allows you to define and manage monitoring for multiple environments (development, staging, production) and services in a consistent manner. 3. **Collaboration**: Engineers and operations teams can collaborate more effectively, as monitoring configurations are treated as code and reviewed through pull requests. 4. **Automation**: Automated checks and alerts can be integrated into CI/CD pipelines, providing real-time feedback on the application's health and performance before it reaches production. 5. **Consistency and Reproducibility:** Monitoring configurations are versioned alongside the application code, ensuring that changes are tracked and can be rolled back if necessary. ## Introducing Checkly’s CLI The [Checkly CLI](https://www.checklyhq.com/cli/overview) gives you a JavaScript/TypeScript-native workflow for coding, testing and deploying synthetic monitoring at scale, from your code repository. Getting started is pretty easy. You can be up and running in no time to ensure your crucial web apps and sites are performing up to spec. The Checkly CLI provides two main workflows: * **Coding**: These encompass scripts (such as ApiCheck, BrowserCheck, or SlackAppAlertChannel) written in JavaScript/TypeScript. They are intended to be deployed and executed on the Checkly cloud backend. * **Command**: These constitute the fundamental commands for executing your monitoring scripts. The `test` command is utilized for running monitoring checks locally or in continuous integration, while the `deploy` command is employed to push your monitoring scripts to the Checkly cloud backend. Let's jump into the step-by-step process. ## Getting Started with Checkly’s CLI ### 1. Installation We’re going to pretend we are working on adding a feature to a web application that also requires some updates to our API backend. We will assume we already bootstrapped our repository with a Checkly CLI project using: ```bash install-checkly-cli theme={null} npx checkly init ``` This command sets up all the basics to kickstart your MaC workflow in your repo. starting the MaC workflow in your repo

In your project directory, you will find a folder named “**checks**” containing the following check templates: ```bash project-structure theme={null} |__checks__ |- api.check.ts |- heartbeat.check.ts |- homepage.spec.ts ``` Once this setup is complete, log in to your Checkly account via the CLI using the following command: `npx checkly login` You can choose to log in from the browser or in your terminal. After logging in, you'll be able to update Checkly Checks from your local machine as long as you're connected to the internet. ### 2. Write Your First Check In your development environment, write JavaScript/TypeScript tests for your code updates, similar to unit tests. We typically use the Playwright testing framework in the .spec.ts or .check.ts file. Consider a scenario where you want to monitor the title of the Checkly documentation and take a screenshot of the page. To do this, replace the code in the homepage.spec.ts with the following: ```jsx homepage.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('Checkly Docs', async ({ page }) => { const response = await page.goto('https://www.checklyhq.com/docs/browser-checks/'); // Ensure the page is loaded successfully expect(response?.status()).toBeLessThan(400); // Check if the page title is as expected const pageTitle = await page.title(); const expectedTitle = 'Introduction to Checkly | Checkly'; expect(pageTitle).toBe(expectedTitle); // Optionally, you can take a screenshot if needed await page.screenshot({ path: 'homepage.jpg' }); }); ``` *** This test uses the page.goto method to navigate to the specified URL (`https://www.checklyhq.com/docs/browser-checks/`). The method returns a response object, which is stored in the response variable. Then we use the expect function to assert that the HTTP status code of the response is less than 400. This is a way to ensure that the page is loaded successfully without any HTTP errors. page.title() retrieves the title of the page and compares it with the expected title ('Introduction to Checkly | Checkly') using the expect function. This ensures that the page title matches the expected value. Finally, we take a screenshot of the page and save it as 'homepage.jpg'. ### 3. Running Test Sessions Now that we have our test scripts ready, let's execute them. Run the following command in the terminal of your project repository to test your checks against Checkly's infrastructure: `npx checkly test` You can review the test session with full logging, videos and traces in the Checkly web interface. Here is the result of the test we just executed: test sessions result

There are also links to the detailed summary of the test at the end of the result in the terminal. Here is an example of the test summary:

If you check the detailed summary, we should have a passed test too: passed test screenshot

### 4. Deploy Your Check Now that you've reviewed and updated your tests, you can proceed to deploy your MaC workflow and related resources, such as alerts and dashboards. Run the following command in your project terminal to deploy the tests to your Checkly account: `npx checkly deploy` Once the deployment is complete, you'll see a success message in your terminal, indicating that the project has been deployed to your Checkly account. To verify this, navigate to the home section on the left side of the Checkly UI, and you'll find the project with the name of the test script from your local repository. checkly dashboard

### 5. Setting up Alerts Checkly offers alert services to notify you whenever a check fails. Various alert channels are available, including Slack, SMS, webhook, phone call, email, Opsgenie, PagerDuty, etc. To set up alerts for your check, go to the specific project, in this case, "homepage.spec.ts." At the top right corner of the code editor, click the "Settings" button. In the revealed side panel, access "Alert Settings" under "Retries & alerting." setting up alerts with checkly

Here, configure monitoring parameters according to your needs, including check frequency, locations, retries and alerting. You can also set up your preferred alert channel using the Checkly CLI. Learn more about the alert channels from the [official documentation](/docs/constructs/alert-channel). With the appropriate alert channels set up, there is no need for customers to regularly visit the dashboard. Instead, they will be promptly notified, allowing them to react immediately upon receiving alerts. ## Integrating Monitoring as Code into Your CI/CD Pipeline with Checkly As you just saw, with Checkly’s CLI, you can run your checks directly from your CI/CD pipeline, using them as end-to-end (E2E) tests. This strategy allows you to validate your application and infrastructure before deploying to production, staging, or any other environment. Additionally, you can manage the entire lifecycle of your checks—creating, updating, and deleting—right from your codebase as part of your CI/CD workflow. Here’s how to integrate your checks and the Checkly CLI if you’re already using one of the popular CI/CD platforms: * [**GitHub Actions**](/docs/integrations/ci-cd/github/actions): Run the Checkly CLI from GitHub Actions, export summary reports, and integrate with monorepos for streamlined monitoring across multiple projects. * [**GitLab CI**](/docs/integrations/ci-cd/gitlab/overview): Use GitLab CI pipelines to run the Checkly CLI, setting up separate jobs for E2E testing and deployment. * [**Jenkins**](/docs/integrations/ci-cd/jenkins/overview): Integrate the Checkly CLI into a Jenkins pipeline using a Jenkinsfile to automate your monitoring checks and deployments. ### CI/CD Integration Using Vendor Webhooks For platforms like [Vercel](/docs/integrations/ci-cd/vercel/overview) or [GitHub deployments](/docs/integrations/ci-cd/github/deployments), you can trigger your Checkly checks using vendor-specific webhooks. This method provides a streamlined way to initiate monitoring as part of your deployment process without needing a fully-fledged CI/CD pipeline. By integrating Checkly’s monitoring checks into your CI/CD pipeline, you ensure that your application is thoroughly validated before and after deployment, reducing the risk of issues in production and maintaining a high level of reliability. ## The Most Reliable Way of Monitoring Monitoring as Code (MaC) simplifies how you manage your monitoring setups by leveraging automation, scalability, and version control. With Checkly's CLI, implementing MaC is straightforward, allowing you to define and manage your monitoring configurations directly from your codebase. By following the steps in this guide, you can integrate Monitoring as Code into your projects, ensuring your applications are consistently monitored, reliable, and ready to meet the demands of modern software development. Get started with Checkly's CLI today to automate and scale your monitoring, and start detecting issues faster. # Monitoring for Broken Links with Playwright & Checkly Source: https://www.checklyhq.com/docs/guides/how-to-monitor-broken-links Learn how to check for broken links on your webpages with Playwright. ## Why you should monitor for broken links. If your content lives in markdown and is only updated on deployments, checking for broken links in CICD will work out fine. But in my experience, this is rarely the case. Website content often comes from a CMS or results from data stored in a database so that people can change content without relying on a developer. And that's great for the team, but it also means that links can change at any time. Your site can and will include broken links at any time. Will this be a big deal? It depends. But if you consider being in charge of a popular newspaper website, a broken top story link on the home page will get someone into trouble. In this case, you can adopt [synthetic monitoring](https://www.checklyhq.com/product/synthetic-monitoring/), constantly run your Playwright scripts on a schedule, and receive alerts when your tests fail. When then someone typoed a broken link into your site, you'll be the first to know about it because you received a timely alert. Run playwright tests on a Schedule with Checkly

Run playwright tests on a Schedule with Checkly

# Creating a Keyword Monitor Source: https://www.checklyhq.com/docs/guides/keyword-monitoring With Checkly, we can use Playwright to create monitors that verify page content by checking for keywords. While basic uptime monitoring tells you if your site is running, it can't tell you if your most critical content is actually visible to customers. Monitoring for keywords or content in a user-interface can help extend coverage and improve your UX and bottom-line. It can be used for several purposes: 1. Content Monitoring: Checking if specific text appears or disappears from web pages 2. Error Detection: Looking for error messages or specific warning text 3. Compliance: Ensuring required disclaimers or legal text remains present 4. Competition Tracking: Monitoring competitor websites for specific product names or features With Checkly, we can use Playwright to create monitors that verify content in these situations. Picture this: you’ve got a successful eCommerce book store. Your best-sellers provide you with 100% of your operating revenue, and your long tail of products provides pure profit. Life is good. But suddenly users start reporting that they came to your shop looking for your best selling book, but couldn’t find it. You only have scattered reports, but by the time you’re investigating your sales, and your profits, are already slipping. Game over. To prevent this disaster scenario, we want to monitor our site constantly to make sure that our best-selling items are always visible, no matter how the site is updated. Tl;dr this is the playwright code to check for a keyword on your page ```jsx Playwright.test.ts theme={null} await expect(page.getByText('KEYWORD')).toBeVisible() ``` If you want to see a real-world example, and how to handle cases like the keyword appearing multiple times, or allowing some variations, read on! ## Creating a keyword assertion in Playwright In this example from the [Danube Web Shop](https://danube-webshop.herokuapp.com/), we need to make sure that ‘Haben oder haben’ is always visible. It’s not enough for the page to load, we need our best seller displayed at the top, so we’ll create an assertion looking for the keyword. An ecommerce storefront

The code for this page monitor reads as follows: ```tsx BestSeller.test.ts theme={null} const { expect, test } = require('@playwright/test') test('Check for my best seller', async ({ page }) => { const response = await page.goto('https://danube-webshop.herokuapp.com/') await expect(page.getByText('Haben oder haben')).toBeVisible() await expect(page.getByText('9.95').first()).toBeVisible() // Take a screenshot await page.screenshot({ path: 'screenshot.jpg' }) }) ``` This requires that an element somewhere on the page have the text `KEYWORD`. It makes no expectation of the element’s type, so it could be a button or a div or a span or anything else. Further, this doesn’t require that the complete text of the element be this, it could be a button that says ‘click here for Keyword’ and this test would pass. As a final note, `getByText` isn’t case sensitive so ‘Keyword,’ ‘KEYWORD’ or ‘KeYwOrD’ would all pass. ## Configuring a keyword monitor in Checkly Next, we’ll add this code to Checkly and set it up to monitor on a set frequency. We have two options for how to set up a page monitor with Checkly: 1. Through the Checkly Web UI 2. As a direct deployment with the Checkly CLI, implementing the [monitoring as code](https://www.checklyhq.com/guides/monitoring-as-code/) model While more advanced operations engineers and QA teams will probably prefer the second option, this guide will show you how to do everything you need from the web UI. If you’d like to know how to configure and deploy this monitor from the command line, see our [documentation on the Checkly CLI](https://www.checklyhq.com/cli/overview). ### Creating a page monitor in the Checkly web interface With our test code from the section above, we can log in to checkly and create a new browser check, select an empty template, and paste in our code. The checkly web UI with our keyword monitoring code

The checkly web UI with our keyword monitoring code

Next we’ll make some decisions about how frequently we want to run our check, under ‘Settings’ for our browser check. * Since my ecommerce shop sees dozens of sales per minute, I’m going to set the frequency of runs to every 2 minutes. * Since I have users in the US and Europe, I’ll select geographic locations there to run checks from. * The failures I’ve experienced are rarely localized to one region, so I’ll select ‘round robin’ to run one check every 2 minutes from one of my geographic locations. The final settings on frequency and locations will look like this: the checkly web UI with multiple frequency settings

the checkly web UI with multiple frequency settings

Next I’ll make a few settings about how we want to retry our checks. * A fleeting failure isn’t a big issue to me. If a single page load didn’t have the best seller for some reason, that’s fine as long as it’s there when you check again a minute later, so I’ll set the retry time to ‘linear,’ retrying 2 more times with a minute between them * Finally I’ll use the global notification settings, and get notified via email, slack, SMS and a via a webhook to [Rootly](/docs/integrations/incident-management/rootly) The checkly web UI with multiple notification settings, described in the article text above

The checkly web UI with multiple notification settings, described in the article text above

With these settings we can rest easy knowing that our users are seeing our best-selling items are visible for all visitors to our main page. ### A complete code example to monitor for a keyword on your page. The code above lacks the wrapper code to be run as a Checkly monitor, here’s a bare-bones example of that code: ```jsx KeywordMonitor.test.ts theme={null} const { expect, test } = require('@playwright/test') test('Check for an element with text containing a keyword', async ({ page }) => { await page.goto('https://www.mysite-url.com/') await expect(page.getByText('keyword')).toBeVisible() // Take a screenshot of the current page await page.screenshot({ path: 'screenshot.jpg' }) }) ``` If you’re unfamiliar with [Playwright](https://www.checklyhq.com/learn/playwright/what-is-playwright/), you may be wondering why there’s no `wait` step after loading the page. Here as in so many page checks we benefit from [Playwright’s auto-waiting](https://www.checklyhq.com/learn/playwright/navigation/#waiting) feature, meaning the code will wait for the element to become visible, and then continue executing as soon as it does. This has a lot of advantages over any manual wait time, aka [‘hard waits’](https://www.checklyhq.com/learn/playwright/waits-and-timeouts/) that can make your page monitors both flakier and slower. If you’d like to see the benefit of auto-waiting, run the script above from the Checkly editor with the ‘Run Script’ command at the bottom right, then take a look at the trace captured from this check run. The report viewer at the left is also where we can view the screenshot we captured with the last line of our check. The checkly web UI showing a trace

In the trace from this check run, we’ll see individual timings for: * The page load * When our target text was visible The checkly web UI showing a trace

In this case, the page load took 1.8 seconds, and only 33ms later, our best seller was visible. At that point the test was passed and we were able to move forward with our check. This saves execution time compared to a manually added wait! ### Narrowing text requirements: exact matching of keywords If you want to find an element that has an *exact* match of the text we searched for, this is an option with the Playwright `getByText` method. ```jsx KeywordMonitor.test.ts theme={null} await expect(page.getByText('Keyword!', {exact: true})).toBeVisible() ``` Now the match will require that text is the exact text of the element, e.g. `Keyword!` and matching the capitalization. ### Matching multiple elements with a keyword Taking a look back at our storefront: our demo ecommerce storefront

A number of strings are repeated in a few places (for example the word ‘Fiction’ or ‘\$9.95’) and a keyword check for those using the code above will fail. If we change the assertion to be: ```tsx KeywordMonitor.test.ts theme={null} await expect(page.getByText('9.95')).toBeVisible() ``` We’ll get an error reading something like: `expect.toBeVisible: Error: strict mode violation: getByText('9.95') resolved to 30 elements` This isn’t a bug in Playwright, it’s just saying: ‘you wanted me to test something about an element, I found many elements that matched the parameters you gave me, so I don’t know which one you were talking about.’ For example, how would we evaluate this test if some of the matches were visible, but others weren’t? Since we can’t give a simple binary answer with multiple results, this check will fail. The Checkly report showing a newly failing monitor

The Checkly report showing a newly failing monitor

In our Checkly report for this monitor, we’ll get notified if this code starts matching multiple results, the monitor’s status will go to ‘failing’ and we’ll get a descriptive report about the error message received. In this case we’ll even get a list of the multiple elements that matched our `page.getByText()`. ### Solving `strict mode violation` when matching keywords Tl;dr you can suppress all these `resolved to [number] elements` by only looking at the first result: ```jsx KeywordMonitor.test.ts theme={null} await expect(page.getByText('9.95').first()).toBeVisible() ``` When I’m writing my own page monitors with Checkly and Playwright, I often add `.first()` preemptively since I’m really only worried that the element appears *somewhere* and don’t care about multiple matches. However this may not make sense for every page monitor! On the web shop above, the page has a top banner offering a sale. We may want to check if this banner is visible by doing a keyword monitor for the text in the banner, but it wouldn’t be good if this banner was showing up twice. a demo ecommerce store showing an error, with two visible offer banners

a demo ecommerce store showing an error, with two visible offer banners

*If our page looks like this, something has gone wrong!* For page components like banners, CTA’s, and login buttons, it may make sense to have our monitor fail if we have multiple matches. ## Checking for multiple possible keywords In another scenario, we might be checking to make sure that each page has a ‘sign up’ call to action somewhere on the page. We start with an assertion like: ```jsx KeywordMonitor.test.ts theme={null} await expect(page.getByText('sign up')).toBeVisible() ``` But it turns out the marketing team keeps rewriting the call to action on various pages, using ‘create an account’ and ‘register your account’ in various places. We’d like to monitor pages to make sure any variation is present in the page. We might spend a minute exploring something like soft assertions, but that’s really not the use case for soft assertions. No, what we want is something like the logical ‘inclusive OR’ that returns a passing check when any of these possible strings is present. ### Checking for keywords with a regular expression almost any place an assertion or locator is taking a string within Playwright, a regex pattern can also be used. ```jsx KeywordMonitor.test.ts theme={null} await expect(page.getByText(/Sign up|register/)).toBeVisible() ``` note that by default in Playwright, patterns are run without the `/i` config, meaning they will be case sensitive. When I first went to write the example above I used `/sign up|register/` and it failed. Made more confusing since the text on the page has `text-transform:uppercase` 🤣 Regex can be quite useful for detailed matching. For example, if I only want to accept correct capitalization I could write: ```jsx KeywordMonitor.test.ts theme={null} await expect(page.getByText(/[Ss]ign [Uu]p|[Rr]egister/)).toBeVisible() ``` so that ‘sIGn uP’ would also cause the monitor to fail. Of course with regular expressions the sky is the limit for what we can match! ### Checking for multiple conditions with `.or()` Since we can return multiple matches on a Playwright locator with `.or()` the other way to match on multiple keywords is with a check like this: ```jsx KeywordMonitor.test.ts theme={null} const { expect, test } = require('@playwright/test') test('visit page and take screenshot', async ({ page }) => { const response = await page.goto('https://danube-webshop.herokuapp.com/') await expect( page.getByText('sign up') .or(page.getByText('register')) ).toBeVisible() // Take a screenshot await page.screenshot({ path: 'screenshot.jpg' }) }) ``` Note that, since we’re now explicitly giving multiple locators, it would be a good idea to add `.first()` so our test will not fail with multiple matches, and examine the first item found for further check conditions: ```jsx KeywordMonitor.test.ts theme={null} await expect( page.getByText('sign up') .or(page.getByText('register')) .first() ).toBeVisible() ``` ## Putting it all together: complete Playwright and Checkly Code for Keyword Monitoring Let’s cover the case where we want to make sure that users see a ‘sign up’ button on the page we’re checking. ```jsx BestSellerOnFrontPage.test.ts theme={null} //best-seller-on-front-page.spec.ts const { expect, test } = require('@playwright/test') test('Check for my best seller', async ({ page }) => { const response = await page.goto('https://danube-webshop.herokuapp.com/') await expect(page.getByText('Haben oder haben')).toBeVisible() await expect( page.getByText('sign up') .or(page.getByText('register')) ).toBeVisible() // Take a screenshot await page.screenshot({ path: 'screenshot.jpg' }) }) ``` Previously we’ve used the Checkly web UI to make our settings to this page monitor, but as you monitor a production site you’ll want to consider adopting a [monitoring as code](https://www.checklyhq.com/guides/monitoring-as-code/) strategy where your page monitors *and* their configuration are stored in a code repository. In that case the same configuration described in a previous section would be stored in a config file: ```jsx BestSellerOnFrontPage.check.ts theme={null} //best-seller-on-front-page.check.ts import { BrowserCheck, Frequency, RetryStrategyBuilder } from 'checkly/constructs' new BrowserCheck('best-seller-on-front-page', { name: 'Best Seller on front page', activated: true, muted: false, shouldFail: false, runParallel: false, runtimeId: '2024.09', locations: ['us-west-2', 'us-west-1', 'eu-central-1', 'eu-south-1'], tags: [], sslCheckDomain: '', frequency: Frequency.EVERY_2M, environmentVariables: [], code: { entrypoint: './best-seller-on-front-page.spec.ts', }, retryStrategy: RetryStrategyBuilder.linearStrategy({ baseBackoffSeconds: 60, maxRetries: 2, maxDurationSeconds: 600, sameRegion: true, }), }) ``` ## Conclusion Keyword-based monitoring with Checkly and Playwright is a simple but effective way to ensure critical content on your site remains accessible and visible, even as updates or unexpected issues arise. From best-sellers in an ecommerce store to key calls-to-action on landing pages, monitoring for specific text gives you peace of mind that your users are seeing the information they need. By configuring Checkly to run frequent, geographically distributed checks and customizing retry strategies, you can detect and respond to potential issues before they impact your business. Whether you’re working with a simple keyword, exact matches, or complex conditions using; the flexibility of Playwright ensures you can tailor your tests to fit any use case. Combined with a [monitoring as code](https://www.checklyhq.com/guides/monitoring-as-code/) approach, you can maintain consistency and version control over both your tests and their configurations. With these tools in place, you’re well-equipped to keep your site reliable and your customers satisfied. # How to Create an API Monitor using an OpenAPI (Swagger) Spec Source: https://www.checklyhq.com/docs/guides/monitoring-an-openapi-spec OpenAPI and Swagger help users design and document APIs in a way that is readable from both humans and machines. As a consequence, they can also be used to generate the code that will run the specified API - both on the provider and consumer side. Can we leverage this same principle to simplify API monitoring? After a brief first look at OpenAPI and Swagger, this article will show how we can quickly use them to monitor a new or existing API. ## What is the OpenAPI Specification? The [OpenAPI Specification (OAS)](https://spec.openapis.org/oas/v3.1.0) specifies a standard, language-agnostic and machine-readable format to describe a web API in one or more files. Nowadays, it acts as a vendor-neutral standard for describing the structure and behaviour of HTTP-based APIs, and exists as part of the [OpenAPI Initiative (OAI)](https://www.openapis.org/). Following the OAS brings the following advantages: 1. A description of the API that is readable from both humans and machines. 2. The possibility of automatically generating documentation, implementation logic and even mock servers for testing. 3. Early validation of the data flows happening within the API. For those new to the OAS and wanting to get a basic understanding without diving straight into the specification itself, the official [OpenAPI Specification Explained](https://learn.openapis.org/specification/) guide is a great place to start. ## Swagger vs OpenAPI The OpenAPI Specification is the newer, fully open source-driven incarnation of the Swagger Specification maintained by [SmartBear Software](https://smartbear.com). The spec describes, produces, consumes, and visualizes RESTful application and web services. The name [Swagger](https://swagger.io/) today indicates a set of tools, both free and paid, that support users of the OpenAPI ecosystem. ## Exploring the specification The core of our API specification will happen in the OpenAPI root document, normally named `openapi.json` or `openapi.yml`. We can choose between JSON and YAML formats as we like. The bare minimum we need to declare is the OpenAPI version, the basic information about our API, and finally the available endpoints. ```yml OpenAPI.yml theme={null} openapi: 3.0.0 info: title: OpenAPI document for our new Payment API version: 0.0.1 paths: {} # Nothing here yet ``` An API without endpoints is not very useful. We can add our first, together with a `GET` operation: ```yml OpenAPI.yml theme={null} openapi: 3.0.0 info: title: OpenAPI document for our new Payment API version: 0.0.2 paths: # User account balance /balance/{id}: # Retrieve the user's balance on the given account get: ... ``` The next step is to define what the operation will look like. In this case, we will specify some constraints on the `id` parameter, as well as the response. ```yml OpenAPI.yml theme={null} openapi: 3.0.0 info: title: OpenAPI document for our new Payment API version: 0.0.3 paths: # User account balance /balance/{id}: # Retrieve the user's balance on the given account get: summary: Get the user's account balance description: Retrieves the current account balance for the given user. operationId: get-balance parameters: - name: id in: path required: true schema: type: string format: uuid responses: "200": description: "OK" content: application/json: schema: $ref: "#/components/schemas/balance" ``` Notice we are explicitly defining which schemas the parameter and response should conform to. In the case of our response, we are referring to an [externally defined schema](https://swagger.io/docs/specification/using-ref). This is helpful when we need to reuse a schema definition multiple times across our spec file. As we proceed, we might add multiple endpoints, each with one or more operations. ```yml OpenAPI.yml theme={null} openapi: 3.0.0 info: title: OpenAPI document for our new Payment API version: 0.0.4 paths: # User account balance /balance/{id}: # Retrieve the user's balance on the given account get: summary: Get the user's account balance ... /transactions: get: summary: Retrieve transaction ... post: summary: Submit transaction ... ... ``` We can keep on building out our API spec as needed. The more precisely we describe our API in this file, the more useful this specification will become, whether we use it for documentation or code generation purposes. ## Generating documentation from OpenAPI Let's now take a look at a more complex, finished example: the [Swagger PetStore demo](https://petstore.swagger.io/v2/swagger.json). Even if we had written it ourselves, this description of a complete API could be overwhelming if we just tried to parse the file line by line. To first-time users, that could feel even more daunting. A better option would be to use the open-source toolset Swagger offers to generate human-readable, interactive API documentation from the file we already have. Pasting the content of the spec file to [Swagger Editor](https://editor.swagger.io) will produce a preview of our [Swagger UI documentation](https://swagger.io/tools/swagger-ui/). This is helpful for consumers of the API, who will be presented with an orderly documentation page breaking down each endpoint while also allowing users to test out different operations. ## Generating boilerplate code from OpenAPI When writing a new API, we will oftentimes need to produce a certain amount of boilerplate code for both our provider (e.g. a server exposing our API for consumption) and consumer (e.g. an SDK for our API) applications. Once we have our OpenAPI file in place, we can use [Swagger Codegen](https://swagger.io/tools/swagger-codegen) (also available within Swagger Editor) to automatically generate that code for us, saving us precious time and reducing the avenues for human error. The following snippet is an example of the code Swagger Codegen can generate starting from an OAS file. ```js OpenAPI.js theme={null} /** * Invokes the REST service using the supplied settings and parameters. * @param {String} path The base URL to invoke. * @param {String} httpMethod The HTTP method to use. * @param {Object.} pathParams A map of path parameters and their values. * @param {Object.} queryParams A map of query parameters and their values. * @param {Object.} collectionQueryParams A map of collection query parameters and their values. * @param {Object.} headerParams A map of header parameters and their values. * @param {Object.} formParams A map of form parameters and their values. * @param {Object} bodyParam The value to pass as the request body. * @param {Array.} authNames An array of authentication type names. * @param {Array.} contentTypes An array of request MIME types. * @param {Array.} accepts An array of acceptable response MIME types. * @param {(String|Array|ObjectFunction)} returnType The required type to return; can be a string for simple types or the * constructor for a complex type. * @param {module:ApiClient~callApiCallback} callback The callback function. * @returns {Object} The SuperAgent request object. */ exports.prototype.callApi = function callApi(path, httpMethod, pathParams, queryParams, collectionQueryParams, headerParams, formParams, bodyParam, authNames, contentTypes, accepts, returnType, callback) { var _this = this; var url = this.buildUrl(path, pathParams); var request = superagent(httpMethod, url); // apply authentications this.applyAuthToRequest(request, authNames); // set collection query parameters for (var key in collectionQueryParams) { if (collectionQueryParams.hasOwnProperty(key)) { var param = collectionQueryParams[key]; if (param.collectionFormat === 'csv') { // SuperAgent normally percent-encodes all reserved characters in a query parameter. However, // commas are used as delimiters for the 'csv' collectionFormat so they must not be encoded. We // must therefore construct and encode 'csv' collection query parameters manually. if (param.value != null) { var value = param.value.map(this.paramToString).map(encodeURIComponent).join(','); request.query(encodeURIComponent(key) + "=" + value); } } else { // All other collection query parameters should be treated as ordinary query parameters. queryParams[key] = this.buildCollectionParam(param.value, param.collectionFormat); } } } // set query parameters if (httpMethod.toUpperCase() === 'GET' && this.cache === false) { queryParams['_'] = new Date().getTime(); } request.query(this.normalizeParams(queryParams)); ... ``` Depending on how much is described in a given API spec, it is possible to move even further and generate tests, mock servers and more. See [openapi.tools](https://openapi.tools) for an up-to-date list of tools belonging to the OpenAPI ecosystem. ## Generated monitoring checks with Checkly [API monitoring](/docs/learn/monitoring/api-monitoring) is key to ensure that our endpoints are returning the correct results in an acceptable timeframe. If we can generate APIs from a OAS description file, we can also generate monitoring checks for them. Checkly is [a member of the OpenAPI initiative](https://www.openapis.org/about), and allows us to import an OpenAPI spec file and automatically creates one or more checks depending on the amount of information available. All it takes is a couple of clicks. When creating an API check, select the `import from Swagger / OpenAPI` button. screenshot of checkly check api creation openapi import button

screenshot of checkly check api creation openapi import button

That will bring up the import dialog. This is where we give Checkly the URL to the Swagger/OpenAPI specification file. screenshot of checkly check api creation openapi import dialog

screenshot of checkly check api creation openapi import dialog

Before anything is created on our behalf, Checkly will show us a summary of all the checks that will be imported. We can choose to import them all defined checks or just a subset. Notice that we can also select whether to import headers, query params and other settings or just to ignore them for now. screenshot of checkly bulk openapi import

screenshot of checkly bulk openapi import

Once we confirm, the checks will be created for us and will start running according to the schedule we have selected. Based on how comprehensively the file we imported described the API we want to monitor, we might have to tweak certain checks manually before they are fully set up. screenshot of checkly checks - with failures

screenshot of checkly checks - with failures

This usually includes adding missing headers or body contents that might not have been explicitly defined in the imported file. We can open the newly created checks and change any of their settings. screenshot of checkly filled in check body

screenshot of checkly filled in check body

After a small amount of tinkering, or none at all, we should have all our checks up and running. screenshot of checkly checks - all passing

For APIs medium and large, and for any that is already described with an OAS file, this check-generating procedure can: 1. Save us a large amount of time otherwise spent in manual check setup. 2. Reduce the chance for human error in check configuration. 3. Help us ensure the right checks are set up according to our single source of truth. [Higher check coverage](/docs/learn/monitoring/api-monitoring#api-monitoring-best-practices) can ultimately give us higher confidence that our API is working as expected, and basing our monitoring on our OpenAPI specification enables us to get to higher coverage faster - without losing flexibility when building our checks. ## Building on our generated monitors As we look to increase our endpoint coverage across our APIs, we might want to explore the following resources: 1. Learning more about [API monitoring](/docs/learn/monitoring/api-monitoring) basics and best practices. 2. Defining our [API checks as code](/docs/guides/getting-started-with-monitoring-as-code) to scale our setup. 3. Integrating our API monitoring with [E2E monitoring](/docs/guides/end-to-end-monitoring/) for our websites. ## Read More

Understand monitoring as code (MaC) via our Checkly CLI. Learn end-to-end monitoring with puppeteer and playwright to test key website flows. Why should the way we manage monitoring be any different?

# How to Monitor an E-Commerce Application with Playwright Source: https://www.checklyhq.com/docs/guides/monitoring-ecommerce-apps-using-playwright Discover how Monitoring as Code transforms cloud infrastructure management with a hands-on Checkly case study. Start optimizing your workflow today! ## Inspired by Infrastructure-as-Code and E2E-Testing Historically, IT infrastructure has been provisioned manually, both on premise and in the cloud. This presented several challenges, including fragmented workflows, lack of transparency and scalability issues. In response to these problems, the last few years have seen a shift to the Infrastructure-as-Code (IaC) paradigm, in which large-scale systems are declared in configuration files as code. A new generation of tools has emerged to serve this use case, the most notable example of which is HashiCorp Terraform. Terraform provides a CLI workflow allowing users to specify what the final infrastructure setup should look like, and then takes care of all the intermediate steps needed to get there. E2E testing evolved similarly during the last decade during from proprietary algorithms living in tools like HP Quality Center to open-source-based code hosted in your repository next to your application code. Today, cross-functional DevOps teams continuously run automated E2E tests in their CI/CD pipeline instead of isolated QA teams testing new versions of your app for three months before release. ## Monitoring as integral part of your SDLC Setting up monitoring can present some of the same issues as provisioning infrastructure or testing. This becomes apparent when we move past the initial rollout or proof-of-concept and onboard multiple products and/or teams, and see our monitoring setup rapidly grow in scope - along with its maintenance needs, that is. Monitoring as code learns from IaC, Unit testing and E2E testing and brings your monitoring close to your development workflows. How? By having it defined as code, much like you would do with any kind of End-to-End tests. ## Why Monitoring as Code with the Checkly CLI What does one gain when moving from a manual to a monitoring as code approach? The **main advantages of the Checkly CLI** are: 1. **Unite E2E testing & monitoring in one workflow.** No more silos between Dev, QA and Ops. 2. **Programmable, testable, reviewable.** Works with your dev pipeline. From your IDE, via PR to CI. 3. **Native @playwright/test support.** No lock-in, just write standard \*.spec.ts files. 4. **Alerting baked in.** Set alerts for Slack, SMS and many more channels. 5. **Typescript-first.** Fully typed for a stellar developer experience with code completion. 6. **Run in the cloud or on-prem**. Run on the Checkly cloud or in your network using [Private Locations](/docs/platform/private-locations/overview) Watch our launch event to learn more about our CLI: ## Monitoring as Code with Checkly Users who have already used Checkly will be familiar with creating Checks, Check Groups, Alert Channels and other resources through the Checkly UI. Creating, debugging and managing E2E and synthetic monitoring at scale is best done “as code”. Currently, Checkly supports three tools you can use for your monitoring as code (MaC) workflow: 1. [The Checkly CLI.](https://www.checklyhq.com/cli/overview) 2. [Terraform (through our Terraform provider)](/docs/integrations/iac/terraform/overview) 3. [Pulumi (through our Pulumi provider)](/docs/integrations/iac/pulumi/overview) ## Monitoring an e-commerce website - as code How does this all look like in practice? Let's find out by creating a small monitoring setup for our [demo e-commerce website](https://danube-web.shop/). ### Setting up the Checkly CLI A quick way to bootstrap a new project with the Checkly CLI is to run the following command. ```bash Terminal theme={null} theme={null} theme={null} theme={null} theme={null} npx checkly init ``` But for this guide, we will show you the manual process to show you all the structure and general idea behind the setup. For our example we will be creating Browser Checks using [Playwright](https://playwright.dev) scripts we have previously written as part of our Playwright guides. ```ts login.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { // navigate to our target web page await page.goto('https://danube-web.shop/'); // click on the login button and go through the login procedure await page.getByRole('button', { name: 'Log in' }).click(); await page.getByPlaceholder('Email').fill('user@email.com'); await page.getByPlaceholder('Password').fill('supersecure1'); await page.getByRole('button', { name: 'Sign In' }).click(); // wait until the login Welcome back message is shown await page.getByText('Welcome back, user@email.com').isVisible(); }); ``` ```ts search.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { const bookList = [ "The Foreigner", "The Transformation", "For Whom the Ball Tells", "Baiting for Robot", ]; // navigate to our target web page await page.goto("https://danube-web.shop/"); // search for keyword await page.click(".topbar > input"); await page.type(".topbar > input", "for"); await page.click("#button-search"); await page.waitForSelector( ".shop-content > ul > .preview:nth-child(1) > .preview-title" ); // halt immediately if results do not equal expected number let resultsNumber = (await page.$$(".preview-title")).length; await expect(resultsNumber).toEqual(bookList.length); // remove every element found from the original array... for (var i = 0; i < resultsNumber; i++) { const resultTitle = await page.$eval( `.preview:nth-child(${i + 1}) > .preview-title`, (e) => e.innerText ); const index = bookList.indexOf(resultTitle); bookList.splice(index, 1); } // ...then assert that the original array is now empty await expect(bookList.length).toEqual(0); }); ``` ```ts checkout.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { // navigate to our target web page await page.goto('https://danube-web.shop/'); // add the first item to the cart await page.getByText('Haben oder haben').click(); await page.getByRole('button', { name: 'Add to cart' }).click(); // navigate to cart and proceed await page.getByRole('button', { name: 'Checkout' }).click(); // fill out checkout info await page.locator('#s-name').fill('Max'); await page.getByPlaceholder('Surname').fill('Mustermann'); await page.getByPlaceholder('Address').fill('Charlottenstr. 57'); await page.getByPlaceholder('Zipcode').fill('10117'); await page.getByPlaceholder('City').fill('Berlin'); await page.getByPlaceholder('Company (optional)').fill('Firma GmbH'); await page.getByLabel('as soon as possible').check(); // confirm checkout await page.getByRole('button', { name: 'Buy' }).click(); // wait until the order confirmation message is shown await page.getByText('All good, order is on the way. Thank you!! Shop more').isVisible(); }); ``` Let's start off by creating a brand new folder: ```bash create-project theme={null} mkdir checkly-mac-cli-example && cd $_ ``` To keep things easy, we create a subdirectory... ```bash create-checks-directory theme={null} mkdir __checks__ ``` ...and copy all our scripts from above into separate files, for example `login.spec.ts`. ...install playwright ```bash install-playwright theme={null} npm init playwright@latest ``` First, install the CLI. ```bash install-checkly-cli theme={null} npm install --save-dev checkly ``` Built-in TypeScript support requires Checkly CLI `v8` or later. On earlier versions, also install a TypeScript loader with `npm install --save-dev jiti` (preferred) or `npm install --save-dev ts-node typescript`. Next up, we want to create our `checkly.config.ts` file and include the basic Checkly configuration as follows: ```ts checkly.config.ts theme={null} import { defineConfig } from 'checkly' import { Frequency } from 'checkly/constructs' export default defineConfig({ projectName: 'Website Monitoring', logicalId: 'website-monitoring-1', repoUrl: 'https://github.com/acme/website', checks: { activated: true, muted: false, runtimeId: '2022.10', frequency: Frequency.EVERY_5M, locations: ['us-east-1', 'eu-west-1'], tags: ['website', 'api'], alertChannels: [], checkMatch: '**/__checks__/*.check.ts', ignoreDirectoriesMatch: [], browserChecks: { frequency: Frequency.EVERY_10M, testMatch: '**/__checks__/*.spec.ts', }, }, cli: { runLocation: 'eu-west-1', } }) ``` We are ready to run the first tests of our project. Use the CLI to authenticate and pick a Checkly account. Make sure you have signed up for a free account on [checklyhq.com](https://app.checklyhq.com/signup). That is achieved by running: ```bash Terminal theme={null} theme={null} theme={null} theme={null} theme={null} npx checkly login ``` After a few seconds, you should see a similar message to the following: ``` $ npx checkly login ? Do you allow to open the browser to continue with login? Yes ? Which account do you want to use? TestCheckly Successfully logged in as Hannes Lenke Welcome to the Checkly CLI ``` Now it's time to test run your browser checks on Checkly: ```bash run-tests theme={null} npx checkly test ``` Success! You should see a similar message to the following: ``` $ npx checkly test Running 3 checks in eu-west-1. __checks__/checkout.spec.ts ✔ checkout.spec.ts (6s) __checks__/login.spec.ts ✔ login.spec.ts (6s) __checks__/search.spec.ts ✔ search.spec.ts (5s) 3 passed, 3 total ``` ### Deploy your first Checks to Checkly Now that our project has been initialised and we have added some resources, we can deploy our Checks by running `npx checkly deploy`. The CLI will determine all the needed changes to be performed to replicate our monitoring code on Checkly. ``` $ npx checkly deploy ? You are about to deploy your project "Website Monitoring" to account "TestCheckly". Do you want to continue? Yes Successfully deployed project "Website Monitoring" to account "TestCheckly". ``` Logging in to our Checkly account, we will see the dashboard has been populated with our three Checks, which will soon start executing on their set schedules. CLI-deployed checks on checkly dashboard

CLI-deployed checks on checkly dashboard

### Monitoring API correctness and performance Browser Checks are now there to keep us informed on the status of our key website flows. What about our APIs, though? Whether they make up the foundation of our service or they are consumed directly by the customer, we need to ensure our endpoints are working as expected. This is easily achieved by setting up API Check resources. ```ts api.check.ts theme={null} // __check__/api.check.ts import { Frequency, ApiCheck, AssertionBuilder } from 'checkly/constructs' const api = new ApiCheck('webstore-list-books', { name: 'List Books API', locations: ['eu-central-1','us-west-1'], // overrides the locations property frequency: Frequency.EVERY_1M, // overrides the frequency property degradedResponseTime: 5000, maxResponseTime: 10000, request: { method: 'GET', url: 'https://danube-web.shop/api/books', assertions: [ AssertionBuilder.statusCode().equals(200), AssertionBuilder.jsonBody('$.length').equals('30') ] } }) ``` We can now once more run `checkly deploy` to see the new Check on Checkly: CLI-deployed api check on checkly

### Alerting Now that we have our Checks in place, we want to set up alerting to ensure we are informed as soon as a failure takes place. Alert channels can be declared as constructs, just like the Checks. Let's add the following to our `alert-channels.ts` file: ```ts alert-channels.ts theme={null} // alert-channels.ts import { EmailAlertChannel } from 'checkly/constructs' const sendDefaults = { sendFailure: true, sendRecovery: true, sendDegraded: false, } export const emailChannel = new EmailAlertChannel('email-channel-1', { address: 'alerts@acme.com', ...sendDefaults }) ``` We are setting up things so that we are alerted when our Check starts failing, as well as when it recovers. But we still need to decide which Checks will subscribe to this channel, and therefore be able to trigger the alerts. This is done by adding the following inside the resource declaration of each check, e.g.: ```ts api.check.ts theme={null} // __check__/api.check.ts import { Frequency, ApiCheck, AssertionBuilder } from 'checkly/constructs' import { emailChannel } from './alert-channels' const api = new ApiCheck('webstore-list-books', { name: 'List Books API', alertChannels: [emailChannel], locations: ['eu-central-1','us-west-1'], // overrides the locations property frequency: Frequency.EVERY_1M, // overrides the frequency property degradedResponseTime: 5000, maxResponseTime: 10000, request: { method: 'GET', url: 'https://danube-web.shop/api/books', assertions: [ AssertionBuilder.statusCode().equals(200), AssertionBuilder.jsonBody('$.length').equals('30') ] } }) ``` The usual `checkly deploy` will apply the changes on our Checkly account: CLI-deployed alert on checkly

We are now fully up and running with our monitoring as code setup. Our Checks will run on a schedule, informing us promptly if anything were to go wrong. Rapidly getting to know about failures in our API and key website flows will allow us to react fast and mitigate impact on our users, ensuring a better experience with our product. You can find the more details about our CLI on our doc [Checkly cli docs](https://www.checklyhq.com/cli/overview). ### More on MaC with our CLI Expanding our initial setup from here is easy, please follow these links if you want to know more: 1. [Why we built Monitoring as Code blog post](https://blog.checklyhq.com/why-we-built-a-developer-first-typescript-cli-for-monitoring-as-code/) 2. [Deep-dive & live coding session](https://www.youtube.com/watch?v=IcaMW2opx2U) 3. [Checkly CLI Docs](https://www.checklyhq.com/cli/overview) ## Read More

Why should the way we manage monitoring be any different? Learn end-to-end monitoring with playwright to test key website flows. Setup scripts are a fundamental tool to tailor API checks to your own target endpoints.

# How to Monitor an E-Commerce Application Using Terraform Source: https://www.checklyhq.com/docs/guides/monitoring-ecommerce-apps-using-terraform The trend of declaring infrastructure as code has been picking up steam over the last few years, offering a way for DevOps teams to transparently manage and scale cloud infrastructure. Why should the way we manage monitoring be any different? In this article, we address this point and illustrate it with a practical example of Monitoring-as-Code on Checkly. The trend of declaring infrastructure as code has been picking up steam over the last few years, offering a way for DevOps teams to transparently manage, monitor and scale cloud infrastructure. Why should the way we manage monitoring be any different? In this article, we address this point and illustrate it with a practical example of Monitoring-as-Code on Checkly. ## What is Monitoring as Code? Monitoring as Code is a transformative approach to maintaining the reliability and performance of modern web applications by integrating monitoring processes directly into the software development lifecycle. This method is fully compatible with Infrastructure as Code (IaC) and configuration management tools, allowing your monitoring solution to be managed alongside other infrastructure components. ### Key Aspects of Monitoring as Code: 1. Developer-First Monitoring: * Seamless Integration: Checkly allows developers to define and manage their monitoring configurations alongside their application code. This means monitoring scripts and configurations can be version-controlled, reviewed, and tested just like any other part of the codebase. * CI/CD Pipeline Support: With Checkly, you can embed monitoring checks directly into your CI/CD pipelines. This ensures that every deployment is automatically verified against your monitoring criteria, reducing the risk of undetected issues reaching production and enabling hassle-free continuous delivery. 2. Programmatic Configuration: * Infrastructure as Code (IaC) Compatibility: Checkly's API and CLI tools make it easy to programmatically set up and manage monitoring. This aligns with the principles of IaC, allowing teams to define their monitoring infrastructure using code, ensuring consistency and reproducibility across environments. * Programmable Monitoring Checks: Define advanced monitoring checks using JavaScript and TypeScript. This flexibility enables you to create highly specific and complex checks tailored to your application's unique requirements. Let's dive deeper into Checkly's Monitoring as Code solution powered by the Checkly CLI. ## Infrastructure-as-Code (IaC) We've released our brand new CLI and recommend giving it a try for the best [Monitoring as Code experience](/docs/cli/overview). Please visit our [documentation](/docs/cli/cli-vs-terraform-pulumi) if you want to learn more about the pros & cons of the Terraform provider vs. the CLI for code monitoring automation. Historically, IT infrastructure has been provisioned manually, both on-premise and in the cloud. This presented several challenges, including fragmented workflows, lack of transparency, and scalability issues. In response to these problems, the last few years have seen a shift to the Infrastructure-as-Code (IaC) paradigm, in which large-scale systems are declared in configuration files, a method that code monitoring adopts to enhance operational efficiency. A new generation of tools has emerged to serve this use case, the most notable example of which is [HashiCorp Terraform](https://www.terraform.io/). Terraform provides a CLI workflow that allows users to specify the desired final infrastructure setup, handling all the intermediate steps and processes needed to achieve it, embodying the principle of **monitoring as code**. provisioning aws infrastructure - code snippet

provisioning aws infrastructure - code snippet

Terraform can provision infrastructure on many cloud vendors thanks to its provider ecosystem. Each provider maps to the vendor's API, offering resources in a domain-specific language known as [HCL](https://www.terraform.io/docs/language/syntax/configuration.html), a cornerstone for both IaC and code monitoring operations. ## Monitoring the IaC way Setting up monitoring tools and monitoring in general can present some of the same issues as provisioning infrastructure. This becomes apparent when scaling beyond the initial rollout or proof-of-concept phase, as monitoring as code helps manage the growing scope and maintenance needs efficiently. Monitoring-as-Code learns from IaC and brings your monitoring config closer to your application and your development workflows. How? By having it also declared as code, much like you would do with any kind of IT infrastructure. ## Why Monitoring-as-Code What does one gain when moving from a manual process to a Monitoring-as-Code approach? The **main advantages** are: 1. Better scalability through faster provisioning and easier maintenance. 2. Better history and documentation: config files can be checked into source control. 3. Shared monitoring setup visibility (and easier shared ownership) in DevOps teams. ## Monitoring-as-Code with Checkly Users who have just started out will be familiar with creating checks, groups, alert channels and other resources through the Checkly UI. However, the official Terraform provider for Checkly allows for these elements to be declared as code, streamlining the provisioning and deployment of active monitoring setups, a practice central to code monitoring. You can [find the Checkly Terraform provider](https://registry.terraform.io/providers/checkly/checkly/latest) on the official Terraform registry. official Checkly Terraform provider on Terraform Registry

official Checkly Terraform provider on Terraform Registry

## Monitoring an e-commerce website - as code Exploring code monitoring in practice, we set up a small monitoring configuration for our [demo e-commerce website](https://danube-web.shop/) using Terraform and Playwright scripts, showcasing the efficiency, reliability and scalability of monitoring as code. ### Setting up our Terraform project For our example we will be creating browser checks using [Playwright](https://playwright.dev) scripts we have previously written as part of our Playwright guides. ```js login.spec.js theme={null} const { chromium } = require("playwright"); (async () => { // launch the browser and open a new page const browser = await chromium.launch(); const page = await browser.newPage(); // navigate to our target web page await page.goto("https://danube-web.shop/"); // click on the login button and go through the login procedure await page.click("#login"); await page.type("#n-email", "user@email.com"); await page.type("#n-password2", "supersecure1"); await page.click("#goto-signin-btn"); // wait until the login confirmation message is shown await page.waitForSelector("#login-message", { visible: true }); // close the browser and terminate the session await browser.close(); })(); ``` ```js search.spec.js theme={null} const { chromium } = require("playwright"); const assert = require("chai").assert; (async () => { // launch the browser and open a new page const browser = await chromium.launch(); const page = await browser.newPage(); const bookList = [ "The Foreigner", "The Transformation", "For Whom the Ball Tells", "Baiting for Robot", ]; // navigate to our target web page await page.goto("https://danube-web.shop/"); // search for keyword await page.click(".topbar > input"); await page.type(".topbar > input", "for"); await page.click("#button-search"); await page.waitForSelector( ".shop-content > ul > .preview:nth-child(1) > .preview-title" ); // halt immediately if results do not equal expected number let resultsNumber = (await page.$$(".preview-title")).length; assert.equal(resultsNumber, bookList.length); // remove every element found from the original array... for (i = 0; i < resultsNumber; i++) { const resultTitle = await page.$eval( `.preview:nth-child(${i + 1}) > .preview-title`, (e) => e.innerText ); const index = bookList.indexOf(resultTitle); bookList.splice(index, 1); } // ...then assert that the original array is now empty assert.equal(bookList.length, 0); // close the browser and terminate the session await browser.close(); })(); ``` ```js checkout.spec.js theme={null} const { chromium } = require("playwright"); (async () => { // launch the browser and open a new page const browser = await chromium.launch(); const page = await browser.newPage(); const navigationPromise = page.waitForNavigation(); // navigate to our target web page await page.goto("https://danube-web.shop/"); // add the first item to the cart await page.click(`.preview:nth-child(1) > .preview-author`); await page.click(".detail-wrapper > .call-to-action"); await page.click("#logo"); // wait until navigation is complete await navigationPromise; // navigate to cart and proceed await page.click("#cart"); await page.click(".cart > .call-to-action"); await page.click("#s-name"); // fill out checkout info await page.type("#s-name", "Max"); await page.type("#s-surname", "Mustermann"); await page.type("#s-address", "Charlottenstr. 57"); await page.type("#s-zipcode", "10117"); await page.type("#s-city", "Berlin"); await page.type("#s-company", "Firma GmbH"); await page.click(".checkout > form"); await page.click("#asap"); // confirm checkout await page.click(".checkout > .call-to-action"); // wait until the order confirmation message is shown await page.waitForSelector("#order-confirmation", { visible: true }); // close the browser and terminate the session await browser.close(); })(); ``` Let's start off by creating a brand new folder: `mkdir checkly-terraform-example && cd $_` To keep things easy, we create a subdirectory... `mkdir scripts` ...and copy all our scripts from above into separate files, for example `login.spec.js`. Next up, we want to create our `main.tf` file and include the basic configuration as follows: ```terraform main.tf theme={null} variable "checkly_api_key" {} terraform { required_providers { checkly = { source = "checkly/checkly" version = "0.8.1" } } } provider "checkly" { api_key = var.checkly_api_key } ``` We are ready to initialise our project and have the Checkly Terraform provider set up for us. That is achieved by running: `terraform init` After a few seconds, you should see a similar message to the following: ```bash theme={null} ragog@macpro learn-terraform % terraform init Initializing the backend... Initializing provider plugins... - Finding checkly/checkly versions matching "0.8.1"... - Installing checkly/checkly v0.8.1... - Installed checkly/checkly v0.8.1 (signed by a HashiCorp partner, key ID 4E5AC4D95E185A57) ... Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. ... ``` ### Creating our first browser checks In the same file, right below our initial instructions, we can now add resources one after the other. They will be browser checks based on the Playwright scripts we previously stored in the `scripts` directory. Here is what each resource could look like: ```terraform main.tf theme={null} resource "checkly_check" "login" { name = "Login E2E" type = "BROWSER" activated = true should_fail = false frequency = 10 ssl_check = false use_global_alert_settings = true locations = [ "us-west-1", "eu-central-1" ] retry_strategy = { type = "LINEAR" } script = file("${path.module}/scripts/login.spec.js") } ``` ```terraform main.tf theme={null} resource "checkly_check" "search" { name = "Search E2E" type = "BROWSER" activated = true should_fail = false frequency = 15 ssl_check = false use_global_alert_settings = true locations = [ "us-west-1", "eu-central-1" ] retry_strategy = { type = "LINEAR" } script = file("${path.module}/scripts/search.spec.js") } ``` ```terraform main.tf theme={null} resource "checkly_check" "checkout" { name = "Checkout E2E" type = "BROWSER" activated = true should_fail = false frequency = 60 ssl_check = false use_global_alert_settings = true locations = [ "us-west-1", "eu-central-1" ] retry_strategy = { type = "LINEAR" } script = file("${path.module}/scripts/checkout.spec.js") } ``` Now that our Terraform project has been initialised and we have added some resources, we can generate a Terraform plan by running `terraform plan`. Terraform will determine all the needed changes to be performed to replicate our monitoring software configuration on Checkly. In doing so, we will be asked for our Checkly API key, which we can find under our account settings as shown below. Not on Checkly yet? [Register a free account](https://app.checklyhq.com/signup) and enjoy your free monthly checks! screenshot of how to get api key from checkly dashboard

screenshot of how to get api key from checkly dashboard

We can expose this as an environment variable in order to spare developers from having to copy-paste it all the time: `export TF_VAR_checkly_api_key=`. ``` ragog@macpro learn-terraform % terraform plan An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # checkly_check.checkout will be created + resource "checkly_check" "checkout" { + activated = true + degraded_response_time = 15000 + retry_strategy { + base_backoff_seconds = 60 + max_duration_seconds = 600 + max_retries = 2 + same_region = true + type = "NONE" } + frequency = 60 + id = (known after apply) + locations = [ + "eu-central-1", + "us-west-1", ] + max_response_time = 30000 + name = "Checkout E2E" + script = <<-EOT const { chromium } = require("playwright"); ... Plan: 3 to add, 0 to change, 0 to destroy. ``` We can now finally apply our changes with `terraform apply`. We might be asked for one final confirmation in the command prompt, after which we will be greeted by the following confirmation message: ```bash theme={null} ... checkly_check.checkout: Creating... checkly_check.login: Creating... checkly_check.search: Creating... checkly_check.checkout: Creation complete after 3s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] checkly_check.login: Creation complete after 3s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] checkly_check.search: Creation complete after 4s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] Apply complete! Resources: 3 added, 0 changed, 0 destroyed. ``` Logging in to our Checkly account, we will see the dashboard has been populated with data from our three checks, which will soon start executing on their set schedules. terraform-created checks on checkly dashboard

terraform-created checks on checkly dashboard

### Monitoring API correctness and performance Browser checks are now there to keep us informed on the status of our key website flows. What about our APIs, though? Whether they make up the foundation of our service or they are consumed directly by the customer, we need to ensure our endpoints are working as expected. This is easily achieved by setting up API check resources. ```terraform main.tf theme={null} resource "checkly_check" "webstore-list-books" { name = "list-books" type = "API" activated = true should_fail = false frequency = 1 ssl_check = true use_global_alert_settings = true degraded_response_time = 5000 max_response_time = 10000 locations = [ "eu-central-1", "us-west-1" ] retry_strategy = { type = "LINEAR" } request { url = "https://danube-web.shop/api/books" follow_redirects = true assertion { source = "STATUS_CODE" comparison = "EQUALS" target = "200" } assertion { source = "JSON_BODY" property = "$.length" comparison = "EQUALS" target = "30" } } } ``` We can now once more run `terraform plan`, followed by `terraform apply` to see the new check on Checkly: terraform-created api check on checkly

### Alerting Now that we have our checks in place, we want to set up alerting to ensure we are informed as soon as a failure takes place. Alert channels can be declared as resources, just like the checks. Let's add the following to our `main.tf` file: ```terraform main.tf theme={null} resource "checkly_alert_channel" "alert-email" { email { address = "" } send_recovery = true send_failure = true send_degraded = false } ``` We are setting up things so that we are alerted when our check starts failing, as well as when it recovers. But we still need to decide which checks will subscribe to this channel, and therefore be able to trigger the alerts. This is done by adding the following inside the resource declaration of each check, e.g.: ```terraform main.tf theme={null} resource "checkly_check" "login" { name = "Login E2E" type = "BROWSER" activated = true should_fail = false frequency = 10 ssl_check = false use_global_alert_settings = true locations = [ "us-west-1", "eu-central-1" ] retry_strategy = { type = "LINEAR" } script = file("${path.module}/scripts/login.spec.js") alert_channel_subscription { channel_id = checkly_alert_channel.alert-email.id activated = true } } ``` Going through the usual `terraform plan` and `terraform apply` sequence will apply the changes on our Checkly account: terraform-created alert on checkly

We are now fully up and running with our monitoring-as-code setup. Our checks will run on a schedule, informing us promptly if critical anything were to go wrong. Rapidly getting to know about failures in our API and key website flows will allow us to react fast and mitigate impact on our users, ensuring a better experience with our product. You can find the complete setup described in this guide on our [dedicated repository](https://github.com/checkly/guides-monitoring-as-code). ### Expanding our setup As our setup expands, we might want to deploy additional tools to make our lives easier. We could: 1. Iterate over existing Playwright scripts and [create multiple checks while declaring only one resource](https://www.checklyhq.com/blog/scaling-puppeteer-playwright-on-checkly-with-terraform/#iterating-through-scripts-for-shorter-config). 2. [Group checks together](https://www.checklyhq.com/blog/scaling-puppeteer-playwright-on-checkly-with-terraform/#grouping-checks-together) to better handle them in large numbers. 3. [Use code snippets](https://www.checklyhq.com/blog/scaling-puppeteer-playwright-on-checkly-with-terraform/#reducing-code-duplication-with-snippets) to avoid code duplication and reduce maintenance. 4. Move your workflow to [Terraform Cloud](https://www.terraform.io/cloud) to easily collaborate with your team when managing your Monitoring-as-Code configuration. ## Read More

Understand monitoring as code (MaC) via our Checkly CLI. Learn end-to-end monitoring with playwright to test key website flows. Setup scripts are a fundamental tool to tailor API checks to your own target endpoints.

# How to monitor the Stripe customer API with Checkly | API Monitoring Examples Source: https://www.checklyhq.com/docs/guides/monitoring-the-stripe-api Application Programming Interfaces (APIs) are used throughout software to define interactions between different software applications. In this article we focus on web APIs specifically, taking a look at how they fit in the Jamstack architecture and how we can set up API monitoring in order to make sure they don't break and respond fast. jamstack architecture diagram

## The rise of APIs With the rise of the [Jamstack](https://jamstack.org/), the already broadly used web APIs have been brought further into the spotlight and explicitly named as cornerstone of a new way of building web applications. In the Jamstack paradigm, applications rely on APIs returning structured data (JSON or XML) when queried via a build process or Javascript-based frontend. The API calls might be aimed at internal services or at third-parties handling complex flows such as content management, authentication, merchant services and more. An example of third-party API could be [Stripe](https://stripe.com/), which acts as payment infrastructure for a multitude of businesses. Given their importance in this new kind of web application, internal and external APIs need to be tightly monitored, because failures and performance degradations will immediately be felt by the end-user. ## API failures API endpoints can break in a variety of ways. The most obvious examples are: 1. The endpoint is unresponsive/unreachable. 2. The response is incorrect. 3. The response time is too high. All of the above can result in the application becoming broken for the end-user. This applies to internal APIs and, especially in the case of Jamstack applications, to third parties as well. API checks allow us to monitor both by mimicking the end-user's own behaviour. ## API checks If we were interested in just verifying a server or a virtual machine's availability, we could rely on a simple ping/uptime monitoring solution. API monitoring is more fine-grained than that though, as we need to validate functionality and performance on each API endpoint. API checks do exactly that, and they are composed of the following: 1. An HTTP request. 2. One or more assertions, used to specify exactly what the response should look like, and fail the check if the criteria are not met. 3. A threshold indicating the maximum acceptable response time. The more customisable the HTTP request is, the more cases can be covered, for example with authentication, headers and payloads. It is worth noting that in real-world scenarios, requests do not happen in a vacuum: they are often handling data retrieved previously, possibly by earlier API calls. Therefore, some mechanism to gather this data and inject it into the request is often needed. Let's dive in deeper into each point. ### HTTP request for REST APIs There is a large variety of valid requests that a user might make to a given endpoint. Being able to customise all aspects of our test request is therefore fundamental. Key aspects are: 1. [Methods](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods), like `GET`, `PUT`, `POST`, `DELETE`, etc 2. [Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers), like `Accept`, `Authorization`, `Content-Type`, `Cookie`, `User-Agent`, etc 3. [Query parameters](https://developer.mozilla.org/en-US/docs/Web/API/URLSearchParams) swagger api documentation

Essentially, we are trying to craft a complete request for exact endpoint. Not only that, but we might want to have multiple requests set up to cover specific options or negative cases, too. One such case can be where user-specified parameters such as pagination and timeframes might largely change the response. This is exemplified by the `List Customers` method in [Stripe's Customer API](https://stripe.com/docs/api/customers/list?lang=curl), which we can use to query elements in very different ways, such as by just specifying a limit of results or asking for all results linked to a specific creation date. In this case, both of the following cases are worth monitoring: ```sh Terminal theme={null} curl https://api.stripe.com/v1/customers \ -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \ -d limit=3 \ -G ``` ```sh Terminal theme={null} curl https://api.stripe.com/v1/customers \ -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \ -d created=1616519668 \ -G ``` If we chose to set up a call using Javascript, for example, we could achieve the same call as in the first case above using [axios](https://github.com/axios/axios): ```js Stripe.js theme={null} const { default: axios } = require("axios"); const AUTH_TOKEN = Buffer.from(process.env.API_KEY).toString('base64') axios({ method: 'get', url: 'https://api.stripe.com/v1/customers', headers: { 'Authorization': 'Basic ' + AUTH_TOKEN, 'Content-Type': 'application/x-www-form-urlencoded' }, data: 'limit=3' }).then((response)=> { console.log(response.data) }) ``` ### HTTP request for GraphQL APIs So far we have looked at a REST API. Let's now take a look at an example of how to build a request for a [GraphQL](https://graphql.org/) endpoint. We will use [GitHub's GraphQL API](https://docs.github.com/en/graphql) to retrieve information about the latest open issues on [Headless Recorder's repository](https://github.com/checkly/headless-recorder). To achieve that, we make a `POST` request to the API's GraphQL endpoint: `https://api.github.com/graphql`. We also need to authenticate to GitHub, by sending over our [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) via the `Authorization` header. The request body will contain our GraphQL query: ```graphql GraphQL.gql theme={null} query { repository(owner:"checkly", name:"headless-recorder") { issues(last:3, states:OPEN) { edges { node { title url labels(first:3) { edges { node { name } } } } } } } } ``` Putting it all together into a cURL command: ```sh Terminal theme={null} curl https://api.github.com/graphql -H "Authorization: bearer " -d '{"query":"query { repository (owner:\"checkly\", name:\"headless-recorder\") { issues (last:3,states:OPEN) { edges { node{ title url labels(first:3) { edges { node { name } } } } } } } }","variables":{}}' ``` The response should look similar to the following: ```json theme={null} { "data": { "repository": { "issues": { "edges": [ { "node": { "title": "UI truncated when default chrome zoom is more than 100%", "url": "https://github.com/checkly/headless-recorder/issues/161", "labels": { "edges": [ { "node": { "name": "🐛 bug" } }, { "node": { "name": "🐣 good first issue" } } ] } } }, { "node": { "title": "Add support for Java?", "url": "https://github.com/checkly/headless-recorder/issues/162", "labels": { "edges": [ { "node": { "name": "🙏 feature request" } } ] } } }, { "node": { "title": "Default to text selectors for Playwright", "url": "https://github.com/checkly/headless-recorder/issues/167", "labels": { "edges": [ { "node": { "name": "🦾 enhancement" } }, { "node": { "name": "❓ question" } } ] } } } ] } } } } ``` ### Assertions To validate the API response, we should be able to check against 1. Status code 2. Headers 3. Body Let's look at an example: creating a customer via the [Stripe Customer API](https://stripe.com/docs/api/customers/create?lang=curl). Since we are not the API's developers, we are assuming the result we get running call right now is correct and can be used to model our assertions. Let's run the following curl command in verbose mode: ```sh Terminal theme={null} curl -v https://api.stripe.com/v1/customers \ -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \ -d description="My First Test Customer (created for API docs)" ``` Within the lengthy output we find the response (in curl denoted by the "\<" symbol), and in it all the important details we need for our assertions. First, we notice the successful status code: ``` < HTTP/2 200 ``` After that, we can see the headers, which we might want to check for: ``` < content-type: application/json < content-length: 1190 < access-control-allow-credentials: true < access-control-allow-methods: GET, POST, HEAD, OPTIONS, DELETE < access-control-allow-origin: * < access-control-expose-headers: Request-Id, Stripe-Manage-Version, X-Stripe-External-Auth-Required, X-Stripe-Privileged-Session-Required < access-control-max-age: 300 < cache-control: no-cache, no-store < request-id: req_S9P5NqvZXzvvS0 < stripe-version: 2019-02-19 < x-stripe-c-cost: 0 < strict-transport-security: max-age=31556926; includeSubDomains; preload ``` And finally the response body, which we might want to inspect to ensure the right data is being sent back: ``` { "id": "cus_JAp37QquOLWbRs", "object": "customer", "account_balance": 0, "address": null, "balance": 0, "created": 1616579618, [clipped] ``` We could expand on our previous code example by add adding an assertion library, such as [chai's](https://www.chaijs.com/api/assert/) or [Jest expect](https://jestjs.io/docs/expect): ```js Stripe.js theme={null} const { default: axios } = require("axios"); const expect = require('expect') const AUTH_TOKEN = Buffer.from(process.env.API_KEY).toString('base64') axios({ method: 'get', url: 'https://api.stripe.com/v1/customers', headers: { 'Authorization': 'Basic ' + AUTH_TOKEN, 'Content-Type': 'application/x-www-form-urlencoded' }, data: 'limit=3' }).then((response)=> { console.log(response.data) expect(response.status).toBe(200) // 1) assert again status code expect(response.headers['content-type']).toBe('application/json') // 2) assert against header expect(response.data['has_more']).toBe(true) // 3) assert against body }) ``` We are now asserting against all three point mentioned above. We could of course go on and add additional assertions against both headers and body. ### Response time thresholds Having an endpoint return the correct result is only half the battle. It is imperative that the response reaches the user quickly enough not to disrupt any dependent workflow. In the worst case, where the response time exceeds what the end user is prepared to wait, a performance failure is undistinguishable from a functional one. The easiest way to handle this requirement would be to assert that the specific response time be lower than a certain value, or even just set a timeout for our axios request by adding the `timeout: 7500` property in the previously shown request config. Instead of simply asserting against a specific response, we might want to set different thresholds: based on the nature of our service, a 2x slowdown might still leave it in what we define as an operational state, while a 10x one might not. ## API monitoring best practices Now that we are clear on the key requirements for setting up API checks, let's think about [what and how we should monitor](https://www.checklyhq.com/blog/what-is-api-monitoring/). ### Monitor every endpoint We want to monitor every API endpoint our application exposes. Remember that different HTTP methods define different API endpoints. For example: 1. `GET /user/:id` 2. `PUT /user/:id` The above count as two separate endpoints, even though the URL is the same. ### Cover key API parameters Some parameters can change the endpoint's response significantly. We should strive to have separate checks verifying that the endpoint is behaving correctly across different configurations. ### Keep checks focused & independent API monitoring checks must be organised as to minimise the time needed to identify resolve the underlying issue. This means we need to keep our checks focused on a specific case (vs trying to have a single check do many things) and independent from each other (vs building chains of checks that build on top of one another). ## Scheduled global API checks Checkly specialises in API monitoring and allows users to run API checks on a schedule from [global locations](/docs/concepts/locations). We can combine these checks with [custom alerting](/docs/communicate/alerts/overview) to be able to quickly respond and remediate potential API issues. checkly dashboard with API checks

A Checkly API check is comprised of the following components. ### Main HTTP request The most basic building block of Checkly's API check is the main HTTP request. This can be fully configured in its method, URL, parameters and body to fully reproduce a real-world web API call. checkly check creation process

> Our previous GraphQL-based example is also supported, see the `GraphQL` option under the `Body` tab. ### Assertions Assertions allow us to check for every key aspect of the response. A check with one or more failing assertions will enter failing state and trigger any connected alert channel. setting up assertions in an api check

In this example, we are checking against: 1. The status code, expected to be `200`. 2. The id of one of the customers returned as part of the response's JSON body. Here we could assert a specific value, but in this case we are happy with just verifying that the field is not empty. 3. The value of the `Content-Encoding` header, expected to equal `gzip`. ### Response time limits Response time limits enable us to set different thresholds to decide exactly which response time maps to a hard failure, a pass or a degradation. We can use transitions between these states to trigger different kinds of alerts using our preferred channels. setting up response time limits in an api check

setting up response time limits in an api check

### Setup and teardown scripts Checkly is highly programmable and allows users to run scripts before and after the main HTTP request of an API check. programmable setup scripts

Setup scripts run before our check and give us access to properties like the URL, headers and query parameters, enabling us to set up all the prerequisites for a successful request. Some examples could be: 1. Fetching a token from a different API endpoint. 2. Setting up test data on the target system. 3. Formatting data to be sent as part of the request. Teardown scripts run after the request has executed, bur right before the assertions. They are useful for manipulating the response (for example to remove sensitive information) or removing any test data on the target system. ## Improving our monitoring As we increase our monitoring coverage across our APIs, we can also increase the efficacy of our setup by: 1. Importing existing [Swagger/OpenAPI specs](/docs/detect/synthetic-monitoring/api-checks/configuration#import-a-swagger--openapi-specification) or even [cURL commands](/docs/detect/synthetic-monitoring/api-checks/configuration#import-a-curl-request) using built-in functionality. 2. [Defining our API checks as code](/docs/guides/getting-started-with-monitoring-as-code) to scale our setup while lowering maintenance needs. 3. Combining our API checks with [E2E monitoring](/docs/guides/end-to-end-monitoring/) for any website or web app service whose API we might be monitoring. ## Read More

Understand monitoring as code (MaC) via our Checkly CLI. Learn end-to-end monitoring with playwright to test key website flows. OpenAPI and Swagger help users design and document APIs in a way that is readable from both humans and machines.

# The Complete Guide to Migrating from Puppeteer to Playwright Source: https://www.checklyhq.com/docs/guides/moving-from-puppeteer-to-playwright The switch from Puppeteer to Playwright is easy. But is it worth it? And how exactly does one migrate existing scripts from one tool to another? What are the required code-level changes, and what new features and approaches does the switch enable? If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. You can use our [puppeteer-to-playwright](https://github.com/checkly/puppeteer-to-playwright) conversion script to quickly migrate your Puppeteer codebase to Playwright. ## Puppeteer and Playwright today While they share a number of similarities, [Puppeteer](https://pptr.dev) and [Playwright](https://playwright.dev) have evolved at different speeds over the last two years, with Playwright gaining a lot of momentum and arguably even leaving Puppeteer behind. These developments have led many to switch from Puppeteer to Playwright. This guide aims to show what practical steps are necessary and what new possibilities this transition enables. Do not let the length of this article discourage you - in most cases the migration is quick and painless. ### Why switch While a comprehensive comparison of each tool's strengths and weaknesses could fill up a guide of its own (see our [previous benchmarks](https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/) for an example), the case for migrating to Playwright today is rather straightforward: 1. As of the writing of this guide, Playwright has been frequently and consistently adding game changing features (see [below](#new-possibilities-to-be-aware-of) for a partial list) for many months, with Puppeteer releasing in turn mostly smaller changes and bug fixes. This led to a reversal of the feature gap that had once separated the two tools. 2. Playwright maintains an edge in performance in real-world E2E scenarios (see benchmark linked above), resulting in lower execution times for test suites and faster monitoring checks. 3. Playwright scripts seem to run even more stable than their already reliable Puppeteer counterparts. 4. The Playwright community on [GitHub](https://github.com/microsoft/playwright/issues), [Twitter](https://twitter.com/playwrightweb), [Slack](https://aka.ms/playwright-slack) and beyond has gotten very vibrant, while Puppeteer's has gone more and more quiet. ## What to change in your scripts - short version Below you can find a cheat sheet with Puppeteer commands and the corresponding evolution in Playwright. Keep reading for a longer, more in-depth explanation of each change. Remember to add `await` as necessary. | Puppeteer | Playwright | | -------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `require('puppeteer')` | `require('playwright')` | | `puppeteer.launch(...)` | `playwright.chromium.launch(...)` | | `browser.createIncognitoBrowserContext(...)` | `browser.newContext(...)` | | `page.setViewport(...)` | `page.setViewportSize(...)` | | `page.waitForSelector(selector)` `page.click(selector);` | `page.click(selector)` | | `page.waitForXPath(XPathSelector)` | `page.waitForSelector(XPathSelector)` | | `page.$x(xpath_selector)` | `page.$(xpath_selector)` | | `page.waitForNetworkIdle(...)` | `page.waitForLoadState({ state: 'networkidle' })` | | `page.waitForFileChooser(...)` | Removed, [handled differently](https://playwright.dev/docs/input#upload-files). | | `page.waitFor(timeout)` | `page.waitForTimeout(timeout)` | | `page.type(selector, text)` | `page.fill(selector, text)` | | `page.cookies([...urls])` | `browserContext.cookies([urls])` | | `page.deleteCookie(...cookies)` | `browserContext.clearCookies()` | | `page.setCookie(...cookies)` | `browserContext.addCookies(cookies)` | | `page.on('request', ...)` | Handled through [page.route](https://playwright.dev/docs/api/class-page#page-route). | | `elementHandle.uploadFile(...)` | `elementHandle.setInputFiles(...)` | | Tricky file download. | Better [support for downloads](https://playwright.dev/docs/downloads). | Did we forget anything? Please let us know by getting in touch, or [submit your own PR](https://github.com/checkly/docs). ## What to change in your scripts - in depth ### Require Playwright package In Puppeteer, the first few lines of your script would have most likely looked close to the following: ```js Puppeteer.js theme={null} const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); // ... ``` With Playwright you do not need to change much: ```js Playwright.js theme={null} const { chromium } = require('playwright'); (async () => { const browser = await chromium.launch(); const page = await browser.newPage(); // ... ``` Playwright offers cross-browser support out of the box, and you can choose which browser to run with just by changing the first line, e.g. to `const { webkit } = require('playwright');` In Puppeteer, this would have been done through the browser's launch options: ```js Puppeteer.js theme={null} const browser = await puppeteer.launch({ product: 'firefox' }) ``` ### The browser context [Browser contexts](https://playwright.dev/docs/api/class-browsercontext) already existed in Puppeteer: ```js Puppeteer.js theme={null} const browser = await puppeteer.launch(); const context = await browser.createIncognitoBrowserContext(); const page = await context.newPage(); ``` Playwright's API puts even more importance on them, and handles them a little differently: ```js Playwright.js theme={null} const browser = await chromium.launch(); const context = await browser.newContext(); const page = await context.newPage(); ``` Like in Puppeteer, for basic cases and single-page flows, you can use the default context: ```js Playwright.js theme={null} const browser = await chromium.launch(); const page = await browser.newPage(); ``` > When in doubt, explicitly create a new context at the beginning of your script. ### Waiting The auto-waiting mechanism in Playwright means you will likely not need to care about explicitly waiting as often. Still, waiting being one of the trickiest bits of UI automation, you will still want to know different ways of having your script explicitly wait for one or more conditions to be met. In this area, Playwright brings about several changes you want to be mindful of: 1. [page.waitForNavigation](https://playwright.dev/docs/api/class-page#page-wait-for-navigation) and \`[page.waitForSelector](https://playwright.dev/docs/api/class-page#page-wait-for-selector) remain, but in many cases will not be necessary due to auto-waiting. 2. [page.waitForEvent](https://playwright.dev/docs/api/class-page#page-wait-for-event) has been added. 3. Puppeteer's [page.waitForXPath](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagewaitforxpathxpath-options) has been incorporated into \`[page.waitForSelector](https://playwright.dev/docs/api/class-page#page-wait-for-selector), which recognises XPath expressions automatically. 4. [page.waitForFileChooser](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagewaitforfilechooseroptions) been removed (see the [official dedicated page](https://playwright.dev/docs/input#upload-files) and our [file upload example](https://www.checklyhq.com/learn/playwright/testing-file-uploads/) for new usage) 5. [page.waitForNetworkIdle](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagewaitfornetworkidleoptions) has been generalised into [page.waitForLoadState](https://playwright.dev/docs/api/class-page#page-wait-for-load-state) (see the `networkidle` state to recreate previous behaviour) 6. [page.waitForUrl](https://playwright.dev/docs/api/class-page#page-wait-for-url) has been added allowing you to wait until a URL has been loaded by the page's main frame. 7. [page.waitFor(timeout)](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagewaitforselectororfunctionortimeout-options-args) becomes \`[page.waitForTimeout(timeout)](https://playwright.dev/docs/api/class-frame#frame-wait-for-timeout). > This is as good a place as any to remind that `page.waitForTimeout` should never be used in production scripts! Hard waits/sleeps should be used only for debugging purposes. ### Setting viewport Puppeteer's `page.setViewport` becomes `page.setViewportSize` in Playwright. ### Typing While puppeteer's [page.type](https://playwright.dev/docs/api/class-page#page-type) is available in Playwright and still handles fine-grained keyboard events, Playwright adds [page.fill](https://playwright.dev/docs/api/class-page#page-fill) specifically for filling and clearing forms. ### Cookies With Puppeteer cookies are handled at the page level; with Playwright you manipulate them at the BrowserContext level. The old... 1. [page.cookies(\[...urls\])](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagecookiesurls) 2. [page.deleteCookie(...cookies)](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagedeletecookiecookies) 3. [page.setCookie(...cookies)](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-pagesetcookiecookies) ...become: 1. [browserContext.cookies(\[urls\])](https://playwright.dev/docs/api/class-browsercontext#browser-context-cookies) 2. [browserContext.clearCookies()](https://playwright.dev/docs/api/class-browsercontext#browser-context-clear-cookies) 3. [browserContext.addCookies(cookies)](https://playwright.dev/docs/api/class-browsercontext#browser-context-add-cookies) Note the slight differences in the methods and how the cookies are passed to them. ### XPath selectors XPath selectors starting with `//` or `..` are automatically recognised by Playwright, whereas Puppeteer had dedicated methods for them. That means you can use e.g. `page.$(xpath_selector)` instead of `page.$x(xpath_selector)`, and `page.waitForSelector(xpath_selector)` instead of `page.waitForXPath(xpath_selector)`. The same holds true for `page.click` and `page.fill`. ### Device emulation Playwright [device emulation settings](https://playwright.dev/docs/emulation) are set at Browser Context level, e.g.: ```js Playwright.js theme={null} const pixel2 = devices['Pixel 2']; const context = await browser.newContext({ ...pixel2, }); ``` On top of that, permission, geolocation and other device parameters are also available for you to control. ### File download Trying to download files in Puppeteer in headless mode can be tricky. Playwright makes this more streamlined: ```js Playwright.js theme={null} const [download] = await Promise.all([ page.waitForEvent('download'), page.click('#orders > ul > li:nth-child(1) > a') ]) const path = await download.path(); ``` See our [example on file download](/docs/learn/playwright/file-download). ### File upload Puppeteer's [elementHandle.uploadFile](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-elementhandleuploadfilefilepaths) becomes [elementHandle.setInputFiles](https://playwright.dev/docs/api/class-elementhandle#element-handle-set-input-files). See our [example on file upload](/docs/learn/playwright/testing-file-uploads). ### Request interception Request interception in Puppeteer is handled via [page.on('request', ...)](https://pptr.dev/#?product=Puppeteer\&version=v11.0.0\&show=api-event-request): ```js Playwright.js theme={null} await page.setRequestInterception(true) page.on('request', (request) => { if (request.resourceType() === 'image') request.abort() else request.continue() }) ``` In Playwright, [page.route](https://playwright.dev/docs/api/class-page#page-route) can be used to intercept requests with a URL matching a specific pattern: ```js Playwright.js theme={null} await page.route('**/*', (route) => { return route.request().resourceType() === 'image' ? route.abort() : route.continue() }) ``` See our [full guide](/docs/learn/playwright/intercept-requests) on request interception for more examples. > For many of the points in the list above, variations of the same function exist at [Page](https://playwright.dev/docs/api/class-page), [Frame](https://playwright.dev/docs/api/class-frame) and [ElementHandle](https://playwright.dev/docs/api/class-elementhandle) level. For simplicity, we reported only one. ## New possibilities to be aware of When moving from Puppeteer to Playwright, make sure you inform yourself about the many completely new features Playwright introduces, as they might open up new solutions and possibilities for your testing or monitoring setup. ### New selector engines Playwright brings with it added flexibility when referencing UI elements via selectors by exposing [different selector engines](https://playwright.dev/docs/api/class-selectors). Aside from CSS and XPath, it adds: 1. Playwright-specific selectors, e.g.: `:nth-match(:text("Buy"), 3)` 2. Text selectors, e.g.: `text=Add to Cart` 3. Chained selectors, e.g.: `css=preview >> text=In stock` You can even create your own [custom selector engine](https://playwright.dev/docs/extensibility#custom-selector-engines). For more information on selectors and how to use them, see [our dedicated guide](/docs/learn/playwright/selectors). ### Saving and reusing state Playwright makes it easy for you to save the authenticated state (cookies and localStorage) of a given session and reuse it for subsequent script runs. Reusing state can [save significant amounts of time](/docs/learn/playwright/writing-tests#keep-tests-independent) on larger suites by skipping the pre-authentication phase in scripts where it is not supposed to be directly tested / monitored. ### Locator API You might be interested in checking out Playwright's [Locator API](https://playwright.dev/docs/api/class-locator), which encapsulates the logic necessary to retrieve a given element, allowing you to easily retrieve an up-to-date DOM element at different points in time in your script. This is particularly helpful if you are structuring your setup according to the [Page Object Model](https://martinfowler.com/bliki/PageObject.html), or if you are interested to do start doing that. ### Playwright Inspector The [Playwright Inspector](https://playwright.dev/docs/codegen#generate-tests-with-the-playwright-inspector) is a GUI tool that comes in very handy when debugging scripts, allowing you to step instruction-by-instruction through your script to more easily identify the cause of a failure.

The Inspector also comes in handy due its ability to suggest selectors for page elements and even record new scripts from scratch. ### Playwright Test Playwright comes with its own runner, [Playwright Test](https://playwright.dev/docs/intro), which adds useful features around end-to-end testing, like out-of-the-box parallelisation, test fixtures, hooks and more. ### Trace Viewer The [Playwright Trace Viewer](https://playwright.dev/docs/trace-viewer) allows you to explore traces recorded using Playwright Test or the BrowserContext Tracing API. Traces are where you can get the most fine-grained insights into your script's execution. playwright trace inspection

### Test Generator You can use the [Playwright Test Generator](https://playwright.dev/docs/codegen) to record interactions in your browser. The output will be a full-fledged script ready to review and execute. page being inspected with playwright codegen

page being inspected with playwright codegen

## Switching to Playwright for richer browser check results Checkly users switching to Playwright can take advantage of its new Rich Browser Check Results, which come with [tracing and Web Vitals](/docs/detect/synthetic-monitoring/browser-checks/performance-metrics) and make it easier to isolate the root cause of a failed check and remediate faster.

performance and error tracing check results on checkly

This reveals additional information about the check execution, including: 1. Overview of all errors raised (console, network and script errors) 2. A timeline summarising the execution across page navigations 3. For each page visited, a network & timing timeline, Web Vitals, console and network tabs. 4. In case of a failing check, a screenshot on failure. > Aside from running a Playwright script, performance and error tracing also require the use of [Runtime](/docs/platform/runtimes/overview) `2021.06` or newer. > Note that cross-browser support is not available on Checkly - [our Browser checks run on Chromium](/docs/detect/synthetic-monitoring/browser-checks/overview) only. ## Read More

Understand monitoring as code (MaC) via our Checkly CLI. Learn end-to-end monitoring with puppeteer and playwright to test key website flows. OpenAPI and Swagger help users design and document APIs in a way that is readable from both humans and machines.

# Guides to Using Checkly Source: https://www.checklyhq.com/docs/guides/overview Comprehensive collection of practical guides for monitoring, testing, and automation with Checkly Welcome to the Checkly Guides section! Here you'll find practical, hands-on guides that help you implement monitoring solutions, automate testing workflows, and optimize your applications for reliability and performance. ## Getting Started Guides Start your journey with Checkly using these foundational guides that cover the essential concepts and workflows. Learn how to implement monitoring as code using Checkly's CLI for scalable, version-controlled monitoring Discover how Checkly integrates into developer workflows and CI/CD pipelines Convert your existing Playwright tests into production monitoring checks ## Monitoring Guides Comprehensive guides for different types of monitoring scenarios to ensure complete coverage of your applications. Learn how to monitor complete user journeys and critical business flows Set up reliable uptime monitoring for your applications and services Monitor specific content and keywords on your websites Implement monitoring throughout your software development lifecycle ## API Monitoring Guides Specialized guides for monitoring APIs and web services with advanced techniques and integrations. Automatically generate monitoring checks from your OpenAPI specifications Set up comprehensive monitoring for Stripe API endpoints and webhooks Create setup scripts to prepare your API environment for testing ## E-commerce Monitoring Guides Industry-specific guides for e-commerce applications covering critical business flows and revenue-impacting scenarios. Monitor critical e-commerce flows like checkout, search, and user authentication Infrastructure as Code approach to e-commerce monitoring ## Playwright Guides Advanced Playwright techniques and best practices for reliable browser automation and testing. Migrate your existing Puppeteer scripts to Playwright Use fixtures to create reusable test data and setup Master Playwright's auto-waiting capabilities for more reliable tests ## Advanced Topics Specialized guides for advanced monitoring scenarios that require deeper technical knowledge. Understand and analyze distributed tracing data for debugging Set up automated monitoring to detect broken links on your websites # How to use environment variables in Checkly Playwright Check Suites Source: https://www.checklyhq.com/docs/guides/playwright-environments Learn how to run the same Playwright tests against different environments and URLs using Checkly. When you set up [a Playwright Check Suite](/docs/detect/synthetic-monitoring/playwright-checks/overview) to run Playwright in the Checkly infrastructure, your tests run in at least two very different contexts. Locally, they hit `localhost`. On Checkly, they hit a deployed URL. While you could maintain separate Playwright config files for each environment, you can also rely on environment variables to handle the entire Checkly workflow from a single `playwright.config.ts`. Run `npx playwright test` locally, [`npx checkly test`](/docs/cli/checkly-test) against a staging URL, and [`npx checkly deploy`](/docs/cli/checkly-deploy) for scheduled production monitoring. ## What Checkly sets for you When your Playwright tests run as a Playwright Check Suite, Checkly sets [environment variables](/docs/detect/synthetic-monitoring/playwright-checks/environment-variables) into your test process: | Variable | Value | Use case | | -------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------- | | `CHECKLY` | `"1"` | Detect that you're running on Checkly at all | | `CHECKLY_RUN_SOURCE` | `"SCHEDULER"`, `"TEST_NO_RECORD"`, `"TEST_RECORD"`, `"TRIGGER_API"`, `"CLI_DEPLOY"`, `"DEPLOYMENT"`, etc. | Distinguish *how* the check was triggered | | `CHECKLY_REGION` | `"eu-west-1"`, `"us-east-1"`, etc. | Know which data center is running the check | | `CHECK_NAME` | String | The name of the check | | `CHECKLY_CHECK_ID` | UUID | The UUID of the check | `CHECKLY` is the simplest variable. It's a boolean flag. `CHECKLY_RUN_SOURCE` gives you more granularity: was this a scheduled production run, a test from the CLI, or a deployment trigger? Please see [the Playwright Check Suites environment variable docs](/docs/detect/synthetic-monitoring/playwright-checks/environment-variables) for all built-in environment variables. You can also define your own variables and secrets at the check, group, or account level. See [Environment Variables](/docs/platform/variables) and [Secrets](/docs/platform/secrets). ## The basic pattern: a simple toggle If you only need two modes (local development and Checkly monitoring), a single boolean defined in your `playwright.config` does the job. ```typescript playwright.config.ts highlight={3} theme={null} import { defineConfig, devices } from "@playwright/test"; const isCheckly = !!process.env.CHECKLY; export default defineConfig({ retries: isCheckly ? 2 : 0, use: { trace: isCheckly ? "on" : "retain-on-failure", }, projects: [ { name: "checkly", use: { ...devices["Desktop Chrome"], baseURL: isCheckly ? "https://staging.example.com" : "http://localhost:3000", }, }, ], webServer: isCheckly ? undefined : { command: "npm run dev", url: "http://localhost:3000", reuseExistingServer: true, }, }); ``` In this scenario, `isCheckly` toggles four things at once: * **`baseURL`**: `localhost` for local dev, your deployed URL on Checkly * **`retries`**: no retries locally (fast feedback), 2 retries on Checkly (resilience against transient network issues and test flakiness) * **`trace`**: always captured on Checkly for debugging, only on failure locally to save disk space (see [Reading traces](/docs/guides/reading-traces)) * **`webServer`**: starts your dev server locally, skipped on Checkly (the deployed site is already running) `baseURL` only takes effect when your tests use relative URLs in `page.goto()` calls, e.g. `page.goto('/')`, not `page.goto('https://example.com')`. This pattern covers most setups. If you have one deployed URL and one local URL, you're done. ## The advanced pattern: environment-aware configuration Some projects need more than two modes. Maybe your CI runs tests against a preview deployment, while Checkly monitors production. Or you want different behavior depending on whether a Checkly run was triggered by a schedule, a deployment, or a manual test. This pattern also works well with [CI/CD integrations](/docs/integrations/ci-cd/overview) where `npx checkly test` runs against preview deployments before promoting to production. Instead of a boolean, you can use `CHECKLY_RUN_SOURCE` to determine which environment your tests are running in and map that to a specific URL. Sometimes it's not just about whether you're running on Checkly or not, but also *where* your tests run. `CHECKLY_REGION` lets you adapt test behavior based on the data center location, for example to validate geo-specific content. ### Step 1: Create an environment helper ```typescript checkly-env.ts theme={null} export type Environment = "DEV" | "CI" | "PROD"; export function getEnvironment(): Environment { // `npx checkly test` sets CHECKLY_RUN_SOURCE to "TEST_NO_RECORD" or "TEST_RECORD" if (process.env.CHECKLY_RUN_SOURCE?.startsWith("TEST")) return "CI"; // Scheduled runs, triggers, and deployments are production if ( process.env.CHECKLY_RUN_SOURCE?.startsWith("TRIGGER") || process.env.CHECKLY_RUN_SOURCE === "SCHEDULER" || process.env.CHECKLY_RUN_SOURCE === "SCHEDULE_NOW" || process.env.CHECKLY_RUN_SOURCE === "GROUP_RUN_ALL" || process.env.CHECKLY_RUN_SOURCE === "CLI_DEPLOY" || process.env.CHECKLY_RUN_SOURCE === "DEPLOYMENT" ) return "PROD"; // No Checkly env vars? We're running locally return "DEV"; } export function getBaseUrl(env: Environment) { switch (env) { case "DEV": return "http://localhost:3000"; case "CI": return "https://preview.example.com"; case "PROD": return "https://www.example.com"; } } export function getLocationCountry() { if (process.env.CHECKLY !== "1") return null; const region = process.env.CHECKLY_REGION; if (region === "eu-central-1") return "DE"; if (region === "eu-west-1") return "IE"; if (region?.startsWith("us-")) return "US"; if (region === "ap-southeast-2") return "AU"; // more regions ... return null; } ``` The logic: `npx checkly test` (what you run in CI pipelines) triggers a `TEST_*` source that maps to your preview/staging URL. Scheduled runs and deployments are production monitoring and point to the live site. Everything else is local dev. Once you know the environment, you can set different `baseURL` values, adjust retry and trace settings, or add other environment-specific logic.

Platforms like Vercel and Netlify expose preview deployment URLs via environment variables (e.g. `VERCEL_URL`). You can pass these to your Checkly test runs using the `-e` flag: `npx checkly test -e ENVIRONMENT_URL=https://preview-abc.vercel.app`.

Your `getBaseUrl` function can then read `process.env.ENVIRONMENT_URL` instead of hardcoding a preview URL. See the [CI/CD integration docs](/docs/integrations/ci-cd/overview) and the [`npx checkly test` reference](/docs/cli/checkly-test) for more details.

### Step 2: Use the helper in your Playwright config ```typescript playwright.config.ts highlight={4-5, 10} theme={null} import { defineConfig, devices } from "@playwright/test"; import { getBaseUrl, getEnvironment } from "./tests/checkly-env"; const environment = getEnvironment(); // "DEV" | "CI" | "PROD" const baseURL = getBaseUrl(environment); // "http://localhost:3000" | "https://preview.example.com" | "https://www.example.com" export default defineConfig({ retries: environment === "DEV" ? 0 : 2, use: { baseURL, trace: environment === "DEV" ? "retain-on-failure" : "on", }, projects: [ { name: "Critical", use: { ...devices["Desktop Chrome"] }, grep: /@critical/, }, ], }); ``` With this approach, you have one config file that targets three environments with different base URLs and no duplication. ### Going further: per-environment behavior in tests If you need per-test access to environment values, you can take the setup from Step 2 further by wiring your helpers into [Playwright test fixtures](https://playwright.dev/docs/test-fixtures). This keeps environment logic out of individual test files. Watch [Reuse Playwright Code across Files and Tests with Fixtures](https://www.youtube.com/watch?v=2O7dyz6XO2s) for a walkthrough of the fixture patterns covered in this section. #### Create a custom test fixture Create a `base.ts` file that extends Playwright's `test` object with custom fixture values. All your test files import `test` and `expect` from here instead of `@playwright/test`: ```typescript tests/base.ts highlight={11,13} theme={null} import { test as base, expect } from "@playwright/test"; import { type Environment } from "./checkly-env"; export type TestOptions = { environment: Environment; locationCountry: null | string; }; export const test = base.extend({ // make `environment` available in your tests environment: ["DEV", { option: true }], // make `locationCountry` available in your tests locationCountry: [null, { option: true }], }); export { expect }; ``` In this example, you specify that `environment` and `locationCountry` should be accessible in your Playwright tests. Their default values can be overwritten via the `use` option in your Playwright config. #### Wire it all up in the Playwright config Pass the helper values into your custom fixtures via the `use` option: ```typescript playwright.config.ts highlight={17-19} theme={null} import { defineConfig, devices } from "@playwright/test"; import { type TestOptions } from "./tests/base"; import { getBaseUrl, getEnvironment, getLocationCountry, } from "./tests/checkly-env"; const environment = getEnvironment(); const baseURL = getBaseUrl(environment); const locationCountry = getLocationCountry(); export default defineConfig({ testDir: "./tests", retries: environment === "DEV" ? 0 : 2, use: { environment, baseURL, locationCountry, trace: environment === "DEV" ? "retain-on-failure" : "on", }, projects: [ { name: "Chromium", use: { ...devices["Desktop Chrome"] }, }, ], }); ``` #### Use fixtures in your tests Your tests stay clean because the environment logic is handled by the fixtures: ```typescript tests/geo-location.spec.ts highlight={4} theme={null} import { expect, test } from "./base"; // `environment` and `locationCountry` are now available in your tests test("geo location works @critical", async ({ page, locationCountry }) => { await page.goto("/"); const locationContainer = page.getByTestId("geo-location"); if (locationCountry) { await expect(locationContainer).toContainText(locationCountry); } else { await expect(locationContainer).toContainText("Geo location unavailable"); } }); ``` The `locationCountry` fixture is automatically populated based on which Checkly region runs the check. There's no need for environment logic in the test itself. ## Wrapping up Start with the simple `isCheckly` toggle. It handles most cases. When you need more than two environments or per-environment behavior beyond `baseURL`, extract the logic into a dedicated helper file. The pattern scales: add a new environment by adding a case to `getEnvironment()` and a URL to `getBaseUrl()`. Your tests and Playwright config stay the same. For the full reference on available environment variables, check the [Playwright Check Suite environment variables docs](/docs/detect/synthetic-monitoring/playwright-checks/environment-variables). # Go From Playwright Testing to Playwright Monitoring with Checkly Source: https://www.checklyhq.com/docs/guides/playwright-testing-to-monitoring Whether you're running a small startup or a large enterprise, integrating Playwright and Checkly into your workflow can transform how you approach testing and monitoring. So, start testing in production today—your users will thank you! We all do testing. We all test code as we write it, and before we release it. But somehow regressions, bugs, and even downtime still happen. Code that worked perfectly on our laptop somehow fails when it meets production, and in ways so obscure that simple integration tests, even canary deploys, don't catch every failure. When we want to ensure reliable software, traditional testing methods often fall short at monitoring real-world scenarios. Enter **Playwright** and **Checkly**, two powerful tools that enable **end-to-end (E2E) testing in production**. In this guide, we'll explore how these tools work together to provide robust monitoring and testing solutions, ensuring your application performs flawlessly for end users. ## The Problem: Monitoring in a Complex World Modern software systems are no longer simple. With the rise of cloud-native technologies, microservices, and distributed systems, applications have become more complex than ever. Front-end frameworks like React and Angular, each implementing modeling that's more complex than an entire 2010 web application, have added further edge cases and wrinkles. Questions like "if this microservice goes down, will users still be able to add items to their cart?" are harder to answer than ever. a diagram showing the ideal SDLC

The result? A flood of monitoring data—logs, metrics, and traces—that can be overwhelming. While observability tools have emerged to help manage this data, they often come with a hefty price tag and can be noisy, making it difficult to extract clear signals about whether your application is truly working. The core question developers and operations teams need to answer is: "Does my app still work?" This means ensuring that the application provides the expected business function correctly and in a timely manner, not just in pre-production environments but in **production**, where real users interact with it. a diagram showing the SDLC with real world feedback

a diagram showing the SDLC with real world feedback

## Why Traditional QA Falls Short Traditional QA processes, including unit tests and end-to-end tests, are essential but often stop at the pre-production stage. Once the application goes live, monitoring takes over, typically handled by a different team using different tools. **This disconnect between QA and monitoring creates gaps in visibility and reliability.** a diagram showing the ideal SDLC

*In the ideal SDLC, Playwright helps you test code pre-deployment, and make sure everything's working* A simple thought experiment is to ask: * Can we agree that no level of pre-deployment QA will catch every failure? * And if so, is it acceptable that everything not caught by QA will have to be found and reported by a user? If your answers to these questions are 'yes' and 'no' respectively, it's time to think about going beyond traditional QA. Testing in production is often treated as a meme, but it shouldn't be. The reality is that production environments are fundamentally different from pre-production environments. Differences in data, security configurations, third-party services, and geographic distribution mean that testing in staging or UAT environments doesn't guarantee success in production. The real dependencies of production apps

The real dependencies of production apps

*In the real world, problems appear at lots of stages, and Playwright can also help you find these edge cases, with the power of Checkly* ## Enter Playwright: A Modern Browser Automation Tool **Playwright** is Microsoft's open-source browser automation framework that simplifies end-to-end testing. It supports multiple platforms and provides a robust API for interacting with web pages and APIs. Here's why Playwright stands out: 1. [**Web-First Locators and Assertions**](https://www.checklyhq.com/learn/playwright/selectors/): Playwright uses web-first locators, making it easier to interact with dynamic web elements without relying on brittle CSS selectors. 2. [**Automatic Retries and Waits**](https://www.checklyhq.com/learn/playwright/waits-and-timeouts/): Playwright automatically retries actions and waits for elements to be visible, reducing flaky tests. 3. [**Powerful Debugging Tools**](https://www.checklyhq.com/learn/playwright/debugging/): Playwright provides traces, videos, and screenshots to help diagnose issues quickly. 4. [**API Testing**](https://www.checklyhq.com/learn/playwright/testing-apis/): Beyond browser automation, Playwright can also handle API testing, making it a versatile tool for end-to-end workflows. 5. [**Detailed Assertions**](https://www.checklyhq.com/learn/playwright/assertions/): No more do we call every `200 OK` response a passing test. Playwright can make complex, programmatic assertions about the responses it receives, checking things like 'every loaded record has a connected ID' or 'all buttons are set to be a brand-approved color.' ## Testing in Production with Playwright and Checkly While Playwright excels at browser automation, **Checkly** takes it a step further, automating runs of Playwright tests and enabling [Monitoring as Code](/docs/concepts/monitoring-as-code). [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) let you run and schedule Playwright tests against production environments, providing real-time insights into your application's health. Run a Playwright test every hour or every minute from locations all across the globe, all managed from a workflow that fits into your current CI/CD model, rather than requiring a cumbersome interface to manage monitors. Checkly Dashboard with multiple Playwright Check Suites

Checkly Dashboard with multiple Playwright Check Suites

### Key Features of Checkly: 1. **Global Test Execution**: Run your Playwright scripts from multiple geographic locations to ensure your application performs well for users worldwide. 2. **Monitoring Cadence**: Run tests every minute or every hour, and set a monitoring cadence that defends your SLA. 3. **Alerting and Notifications**: Set up alerts via email, Slack, PagerDuty, or other channels to notify your team of failures. 4. **Connect Backend Traces**: With Checkly Traces, harness the power of OpenTelemetry to [see backend traces connected to each testing session.](/docs/resolve/traces/overview) 5. **Code Workflow Integration**: Manage your monitoring scripts as code, enabling version control, collaboration, and CI/CD integration. ## A Practical Example: Monitoring an E-Commerce Workflow Let's say you're running an e-commerce platform. Any regression, missing feature, or unavailability means users can't check out on your site, so even a few minutes of a problem costs your business money. A critical user journey might involve: 1. Logging into the application. 2. Searching for a product. 3. Adding the product to the cart. 4. Checking out. A typical web development process would have QA or developers run tests before deployment, and then 'monitor' deployments to make sure the site is still available after releases. Playwright enhances this pre-deployment testing by letting you simulate user behavior in detail. However, when features don't work as expected in production, in a way that wasn't detectable in a QA environment, you might end up relying on your users to tell you. **By the time problems are identified, revenue (and trust) has already been lost.** With Playwright, you can automate the process of testing your site, removing the need for a human to click through your pages. And with Checkly's [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview), you can take your existing Playwright test suite, run it globally and on a cadence, meaning your monitoring will be the first place you'll hear about problems with the critical user paths. Here's how Checkly enhances the process of releasing new code: 1. **Write the Playwright Script**: Automate the login, product search, and checkout steps. Run initial tests through the Checkly network to make sure it's working from all your users' geo-locations. 2. **Configure Monitoring Settings**: Define how often the script should run (e.g., every 15 minutes) and from which locations (e.g., Ireland, Frankfurt). 3. **Set Up Alerts**: Notify your team if a Playwright test fails, ensuring quick resolution of issues. 4. **Deploy to Production**: Use [Checkly's CLI to deploy](/docs/cli/checkly-deploy) your Playwright tests and start monitoring your production environment. With Checkly running your Playwright monitors, you can find and resolve issues before your users are affected. ### What to Consider When Getting Started with Monitoring in Production But what do you need to consider when you start globally monitoring in production? How do you make your Playwright tests geo-location aware? How can you avoid having your tests counted as real users? And should every test become a Playwright monitor? We covered all these questions in a recent webinar. If you want to learn how to apply different Playwright settings depending on the execution environment, watch the recording. This guide will include essential snippets below. ### Environment-Aware Playwright Tests Once your Playwright tests run locally, in CI/CD, and as scheduled monitors, they need to be aware of their execution environment. A local Playwright test executed with `npx playwright test` needs to target `localhost`, a CI/CD test run via [`npx checkly pw-test`](/docs/cli/checkly-pw-test) or [`npx checkly test`](/docs/cli/checkly-test) needs to target environment-specific preview URLs, and global production monitoring might need to be aware of its execution region. To differentiate your Playwright test runs in Checkly, use the [provided environment variables](/docs/detect/synthetic-monitoring/playwright-checks/environment-variables). ```ts environment.ts theme={null} export function getEnvironment(): Environment { // Playwright running via: // $ npx checkly pw-test // $ npx checkly test const isTest = process.env.CHECKLY_RUN_SOURCE?.startsWith("TEST"); if (isTest) return "CI"; const isMonitoring = process.env?.CHECKLY_RUN_SOURCE?.startsWith("TRIGGER") || process.env?.CHECKLY_RUN_SOURCE?.startsWith("SCHEDULE") || process.env?.CHECKLY_RUN_SOURCE?.startsWith("GROUP") || process.env?.CHECKLY_RUN_SOURCE === "CLI_DEPLOY" || process.env?.CHECKLY_RUN_SOURCE === "DEPLOYMENT" || process.env?.CHECKLY_RUN_SOURCE === "SCHEDULE"; if (isMonitoring) return "PROD"; // Playwright running locally via: // $ npx playwright test return "DEV"; } ``` After knowing about your Playwright environment, adjust `baseURL`, `retries`, and any other additional configuration. ```ts base-url.ts theme={null} /** * Evaluate Playwright's `baseURL` depending on the environment */ export function getBaseUrl(env: Environment) { const isDev = env === "DEV"; if (isDev) return "http://localhost:3000"; const isCI = env === "CI"; if (isCI) return process.env.PREVIEW_URL; const isProd = env === "PROD"; if (isProd) return "your-production-site.com"; } ``` ```ts retries.ts theme={null} /** * Evaluate Playwright `retries` depending on the environment */ export function getRetries(env: Environment) { const isDev = env === "DEV"; if (isDev) return 0; const isCI = env === "CI"; if (isCI) return 1; // increase retries to avoid unnecessary monitoring alerts const isProd = env === "PROD"; if (isProd) return 2; return 0; } ``` ```ts blocked-domains.ts theme={null} /** * Evaluate Playwright settings based on the environment */ export function getBlockedDomains(env: Environment) { if (env === "PROD") { return [ "analytics.example.com", "tracking.example.com", "dummy-tracker.js", ]; } return []; } ``` If your site is geo-location aware, use `process.env.CHECKLY_REGION` to adjust your tests based on where your tests are executed. ```ts theme={null} export function getLocationCountry() { if (process.env.CHECKLY !== "1") return null; if (process.env.CHECKLY_REGION?.startsWith("eu-central")) return "DE"; if (process.env.CHECKLY_REGION === "eu-west-1") return "IE"; // more location handling // ... return null; } ``` ### Request Blocking to Avoid Incorrect Analytics If you start running Playwright against production sites, you must ensure not to mess with possible client-side analytics. Luckily, [request interception is quickly done in Playwright](/docs/learn/playwright/intercept-requests). ```ts theme={null} page.route("**/*", (route) => { const blockedDomains = ["analytics.example.com", "tracking.example.com"]; if ( blockedDomains.some((domain) => route.request().url().includes(domain), ) ) { console.log("Blocking:", route.request().url()); return route.abort(); } return route.continue(); }); ``` ### Monitoring Selection via Playwright Projects Playwright test suites running over ten minutes are very common. And while it makes sense to run all tests before going live and deploying to production, a slow test suite is not ideal for monitoring because it will delay sending triggered alerts. **If your Playwright monitoring takes ten minutes to finish, your users might already face production issues for ten minutes before you get notified.** Playwright test selection for testing and monitoring

Playwright test selection for testing and monitoring

To guarantee quick and timely alerts, select only the most critical user journeys to become Playwright monitors. Playwright Check Suites let you [select tests based on Playwright projects and tags](/docs/detect/synthetic-monitoring/playwright-checks/test-organization). ```ts checkly.config.ts highlight={9, 15} theme={null} export default defineConfig({ checks: { playwrightChecks: [ { name: "Playwright Search Monitoring", logicalId: "search-monitoring", // Select the tests defined in // the `Search` Playwright project pwProjects: ["Search"], }, { name: "Playwright Login Monitoring", logicalId: "login-monitoring", // Select tests tagged as `@login` pwTags: ["@login"], }, ], }, }) ``` ```ts playwright-monitor.check.ts highlight={4,16} theme={null} new PlaywrightCheck("search-monitoring", { name: "Playwright Search Monitoring", playwrightConfigPath: "../playwright.config.ts", pwProjects: ["Search"], activated: true, muted: false, shouldFail: false, locations: ["us-east-1", "eu-west-1", "ap-southeast-2"], frequency: Frequency.EVERY_10M, alertChannels: [emailAlert], }); new PlaywrightCheck("login-monitoring", { name: "Playwright Login Monitoring", playwrightConfigPath: "../playwright.config.ts", pwTags: ["@login"], activated: true, muted: false, shouldFail: false, locations: ["us-east-1", "eu-west-1", "ap-southeast-2"], frequency: Frequency.EVERY_10M, alertChannels: [emailAlert], }); ``` ## Solving the Cultural Problem: DevOps in Practice One of the biggest challenges in monitoring and testing is the cultural divide between development and operations teams. By adopting **monitoring as code**, you bridge this gap. Developers can write and maintain monitoring scripts using the same tools and workflows they use for application code. This approach aligns with the core principles of DevOps, where operational problems are solved using development practices. ## Conclusion Testing in production is no longer a meme—it's a necessity. With tools like **Playwright** and **Checkly**, you can automate end-to-end testing, monitor your application in real-time, and ensure it delivers a seamless experience to users. By treating monitoring as code, you empower your team to proactively address issues, reduce downtime, and improve overall reliability. Whether you're running a small startup or a large enterprise, integrating Playwright and Checkly into your workflow can transform how you approach testing and monitoring. So, start testing in production today—your users will thank you! # A guide to diagnosing failures with Playwright Traces Source: https://www.checklyhq.com/docs/guides/reading-traces When you get a notification of a failed check, you can often determine the cause right in the Checkly dashboard. This guide covers some the most common failure modes and where to find answers. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Checkly warns you of failures before they’re reported by users. With well-timed alerts you can find out minutes or hours before you get your first support ticket that something is broken. But once we’ve gotten a notification of a failure, how do we do the first step of root cause analysis: diagnosing the cause of the failure? Browser checks, especially, involve multiple systems that have to ‘work’ to pass a check, how do we find out if network issues, script errors, or a garbled database row, were possible causes? This article shows you how to go from an alert to the first part of repair: finding the error. ## Diagnosing a problem based on the pattern: interpreting a check dashboard Whenever you’re working a technical incident, it’s tempting to go to the most detailed information first: checking traces from individual checks is a bit like starting incident investigation in application logs: the answer is in there, but there’s a lot of information to sift through. How can we look at high-level dashboards showing multiple check runs, and start to diagnose possible causes? To emphasize: none of the patterns mentioned here are proof positive, and you should always take the second step of looking into the check’s traces. That detailed analysis is covered in the second half of this guide. ### Identifying a growing problem When you’re looking at the pattern of failures, one question you’d like to answer is: ‘did \[x] code release cause this failure?’ To answer this quickly, look for a pattern like the one below, where a small problem grows with time: a Checkly dashboard

*In this case it looks like a problem has been getting worse for a while, with slow performance causes ‘degraded’ results before checks stop passing entirely* In this case we can see that degraded results started on or about April 24th. Even though the alerts started a few days later, it’s likely the problem has been growing since late April. Again, this isn’t proof positive since all those results were only ‘degraded,’ a good next step would be to check the resource use of the services. This might be something like a memory leak that’s getting worse with time! ### The comb: failures with set intervals In the chart below, this check is currently passing, but an oddly regular pattern of failures has been happening for some time. This indicates a common failure mode: a Checkly dashboard

*This regular failure that never affects all checks look like it might be a failure in one region.* What would cause some, but not all, checks to fail with some regularity? Checking the list of failure locations, we see that all the checks are coming from one place. a Checkly dashboard

*Sure enough one geographic region is failing.* In this case geo-fencing, localization or some other geographic specific setting is causing only one check location to fail. A few settings to detect this issue more reliably: If you’re often getting failures in one region that’s critical to your business, consider increasing check frequency or setting your checks to ‘parallel runs’ meaning that ever selected geographic region every time a Checkly dashboard

*Parallel vs. round-robin checks, each set with a five-minute interval.* Second, if you’ve selected round-robin checks but still want to detect single-region failures quickly, set your retry strategy to always retry from the same region. a Checkly dashboard

*Now a failure in Cape Town will run all its check attempts from Cape Town* To read further about retries, regions, and timing, check out how to [optimize your MTTD with the right check frequency](https://www.checklyhq.com/blog/check-frequency/). ## Diving in and finding the problem: beyond the video Let’s start with a simple example of a failing test. In this case my web shop’s category view is supposed to look like this: a Checkly dashboard

*A user expects to be able to find top sellers and hidden gems in the category view.* But now, my check on this page is failing, and I’d like to figure out why. The pattern is sudden and complete, with all checks failing, starting a few hours ago. ### We all start by checking the video Reader, I’ll level with you: the purpose of this article is to get you to do more than checking the video on a failed check. But before I offer more routes to check I wish to acknowledge *we all start by checking the video.* A video has the great benefit of showing both the failure and the ‘normal’ behavior of the check up until that point, it reminds us what page we were loading and there’s a good chance that the failure is visual enough to show up right away. a Checkly dashboard

*In this example, the gap between the video and what we expect in the screenshot above is significant, the ‘bottom sellers’ section of the page is missing entirely.* So, of course, we’re all going to check the video capture early in the troubleshooting process. But while the video is nice for finding glaring issues but doesn’t offer much detail beyond that. To know why this page doesn’t look right, we need to go a few steps beyond. ### What loaded and what didn’t: the network tab The network tab is often a good next step when the video didn’t have our answer: without loading the full trace it’s a quick way to see if anything failed to load or loaded very slowly. a Checkly dashboard

*The network tab shows the same details you’d get from a browser’s dev tools* In this case, the only failed kit was a fontawesome library, which seems unlikely to break a page component, so it’s time to move on! ### Check each retry - viewing past attempts If you’re investigating a check with extremely strange or stochastic results, you might wonder ‘this check retried twice before it reported failure, how did those other check runs go?’ We know since we got an alert the other check runs also failed, but to go deeper just use the drop-down on the top left of the check run viewer. a Checkly dashboard

You can select each attempt in the drop-down. ### Try the new AI explainer When trying to interpret a failure, especially during a late night incident response process, it’s helpful to get a summary of how this failure affects users, written in plain English. Translating technical error messages into a more readable format is something that language models excel at, so you may want to try out our AI Impact Analysis tool. For this failure, the analysis looks pretty spot on: users can’t see a section labelled ‘bottom sellers’ a Checkly dashboard

AI Impact analysis is a beta feature, which you can switch on in your account settings. a Checkly dashboard

While the AI explainer is helpful in figuring out the scope of the problem, to go further we really need the big guns: the full trace of this test session. ### Pro tools: the full trace How many times has a user reported a failure somewhere on your site, and you’ve wished they could send a full trace from their dev tools? In some support teams, asking for this browser trace is a common first step in the diagnosis process. For problems detected early with Checkly, the same detailed information is available as soon as you get an alert. Click the trace button to get the whole story. a Checkly dashboard

*The magic happens under the ‘trace’ button.* The full trace will load, let’s step through some of the information on the trace view: a Checkly dashboard

1. The test is broken down into components, and you can scroll through and see what had loaded as each test was happening. 2. Screenshots of the page as loads and formats. You can check a screenshot for each assertion. 3. The network tab that was so useful previously is available here. 4. The console, errors, and log messages from the browser 5. The Call tab which shows exactly how the current playwright call was interpreted. This is so interesting it deserves its own screenshot: a Checkly dashboard

If you notice that the page took a second or two to fully load, and you’re worried it caused the `expect.toBeVisible` call to fail: probably not! Remember that [Playwright’s auto-waiting](https://www.checklyhq.com/learn/playwright/waits-and-timeouts/) ensures that the element will be found as soon as it loads, and your test won’t fail just because of a flash of unformatted content. Traces aren't only for GUIs, either. You can download a trace and analyze it in the terminal, or let a coding agent do it for you — see [Debugging with AI agents](/docs/learn/playwright/debugging-with-ai-agents/). ### **Conclusions** Effective incident diagnosis begins with understanding the patterns in your check failures before diving into detailed traces. Starting with the video replay is natural, as it provides immediate visual context, but the real answers often lie in the network logs, console errors, and full traces. Ultimately, Checkly equips you with the data needed to move from alert to resolution efficiently. By combining pattern recognition with detailed trace analysis, you can diagnose issues faster, reduce downtime, and maintain a seamless user experience. For further optimization, explore [fine-tuning check frequencies](https://www.checklyhq.com/blog/check-frequency/) and retry strategies to minimize detection time—because the sooner you find the problem, the sooner you can fix it. # Reusing Playwright Monitors Across Every Phase of your Software Development Life Cycle Source: https://www.checklyhq.com/docs/guides/sdlc-monitoring The same Playwright code you write for end-to-end testing before deployment, can also test code earlier as part of the dev process, and end-to-end tests can be run on a cadence in Checkly to turn test code into monitoring code. Recently I had a chance to write about going from Playwright testing, [using Playwright to run e2e tests against your staging/pre-deploy environment, and Checkly to monitor your production envrionment](https://www.checklyhq.com/guides/playwright-testing-to-monitoring/). Once you’re using the power of Playwright and Checkly together to monitor your production environment, it’s time to go one step further: use the powerful tools from Checkly to monitor all your environments, from local development on your laptop, to a staging environment, to production. With this guide, the same code you wrote in Playwright for your initial tests on your laptop will get re-used (as appropriate) to monitor Staging, and finally Production to let you know that everything’s working, and find problems before your users do. ## The Problem: A Fractured Picture of Your Code If you’re already excited about Playwright and/or Checkly, you probably don’t need much convincing that it’s good to use both to monitor more environments, if so feel free to skip this section. In short, we’re all aware that the software development life cycle (SDLC) often breaks down when it comes to finding bugs and solving outages. a diagram showing the ideal vs the real SDLC

a diagram showing the ideal vs the real SDLC

Ideally development should only be getting requirements from planning and design, with any feedback (issues that need fixing) coming in the QA stages before deployment to production. In reality a huge list of failures can result in developers having to fix failures and re-engineer code to keep production running. There are two significant concerns that a more unified testing and monitoring approach can help us address: 1. When bugs are found post deployment, feedback is often given without full context, with the first reports going to someone other than the developer. 2. Any fixes have to be delivered out-of-band of the normal development cycle, either coded in haste during [incident response](https://www.checklyhq.com/learn/incidents/what-is-incident-response/), or written and deployed during time that was supposed ot be spent on new features. By unifying around Checkly, **feedback is delivered directly to developers**, with developers working on their own tests, which then offer direct feedback with all the context that product teams need to diagnose root causes. The many unforeseen failures caused by the interaction of application code and the production environment should be monitored with **the same quality of tests that scruitinized our code pre-deployment.** ## The Solution: Unified Checkly & Playwright Monitoring Why do we need to test after staging? After all, if a bug gets through to production, it just means we need to do more testing and better testing before production right? a diagram showing the ideal vs the real SDLC

In reality no matter how much effort we put into simulating a production environment in our pre-deployment testing (hereafter we’ll refer to the environment before production as ‘Staging’ even though your team might call it something else), there are always large gaps between our testing and production environments. Things like third-party API’s, the real resource requirements of production data, and users abilities to find edge cases where we thought none existed, all serve to keep us well supplied with failures that only happen on production. All of this doesn’t mean the work we did on testing before deployment was without value. Local testing gives the fastest possible feedback, often in seconds. And testing on Staging provides a consistent environment with detailed feedback. a diagram showing the ideal vs the real SDLC

Each phase of the SDLC is a valuable place to run testing. What Checkly adds to the equation is running in-depth end-to-end tests on a cadence, meaning you can monitor your own production app as thoroughly as you do pre-deploy code. How does Checkly integrate throughout the process? The map looks something like this: a diagram showing Checkly integrated in dev, deploy, and monitor phases of code diaployment

a diagram showing Checkly integrated in dev, deploy, and monitor phases of code diaployment

*In this diagram, Checkly uses the power of Playwright and OpenTelemetry to give you insight into how your code is running at every stage of the software development life cycle* Checkly serves as both a runner of your Playwright tests throughout the process, and place to see the results of your tests (Playwright Traces) and to collate your test results with backend data (through Checkly Traces, more on this in section 2 below). The result is a process that’s more integrated, that finds problems more proactively, and involves developers in the observability process from day 1. ## 1. Local testing - Playwright on the laptop In the first phase, our test build is running on our local workstation, and we want test feedback as quickly as possible. The few seconds it takes to run a basic Playwright test might be the most time acceptable, if a developer is checking her test feedback every time she writes a few lines of code. ### Local testing with Playwright The process starts with developers working with their test code at the same time they’re writing features. Some devs may use tests written by QA or other teams, and should make use [test fixtures](https://www.checklyhq.com/blog/how-to-implement-custom-test-fixtures-in-playwright/) to automate repeated tasts within their Playwright tests. ### Example: Creating a Custom Fixture for Logging In Let’s say you want to log in to your application before running a set of tests. Instead of repeating the login code in every test, you can create a custom fixture that handles the login process. **Step 1: Define the Custom Fixture** Create a `fixtures.js` file to define your custom fixture. This fixture will extend the base `test` object provided by Playwright. ```ts Playwright.test.ts theme={null} // fixtures.js import { test as base } from '@playwright/test'; // Define a custom fixture that includes a logged-in page export const test = base.extend({ loggedInPage: async ({ page }, use) => { // Perform login steps await page.goto(''); await page.fill('input#username', 'testuser'); await page.fill('input#password', 'password123'); await page.click('button#login'); // Wait for the login to complete (e.g., redirect to the dashboard) await page.waitForURL(''); // Pass the logged-in page to the test await use(page); }, }); export { expect } from '@playwright/test'; ``` **Step 2: Use the Custom Fixture in Tests** Now that the fixture is defined, you can use it in your tests. Import the custom `test` object from `fixtures.js` and write your tests as usual. ```ts Playwright.test.ts theme={null} // example.spec.js import { test, expect } from './fixtures'; test('Test 1: Verify dashboard loads', async ({ loggedInPage }) => { // Use the logged-in page provided by the fixture await expect(loggedInPage).toHaveTitle('Dashboard'); await expect(loggedInPage.locator('h1')).toHaveText('Welcome, testuser!'); }); test('Test 2: Verify user profile link', async ({ loggedInPage }) => { // Navigate to the profile page await loggedInPage.click('a#profile-link'); await expect(loggedInPage).toHaveURL(''); await expect(loggedInPage.locator('h1')).toHaveText('User Profile'); }); ``` *** With test fixtures you can share test code across your team, saving them time as they write local tests. ### Test your laptop from all over the globe with the Checkly CLI After you’ve done the first extremely-high-speed iterative testing while writing new code, you may want to see how your local build handles requests sent from one or more real-world locations. This would require that your build is available to the internet either through a dev server or a reverse proxy tool like [ngrok](https://ngrok.com/). With Checkly you can write your Playwright tests and then run them through the Checkly system, running your test from multiple geographic locations right from your local terminal with the [Checkly CLI](https://www.checklyhq.com/cli/overview). After setting up your tests in a Checkly repisitory, a simple `npx checkly test` will run your Playwright tests, not from your laptop but from the geographic locations defined in your `checkly.config.ts` file. The Checkly CLI running playwright tests from the Checkly network

The Checkly CLI running playwright tests from the Checkly network

*With the Checkly CLI your tests run from all the geographic locations in your configuration.* After you’re happy with your Playwright tests run through Checkly, you’ll be ready to deploy your tests as monitors to the checkly system, and store that code in a shared repository. All of this is part of the monitoring as code standard that Checkly supports. More on this in the following sections. ## 2. Staging - Monitoring pre-deploy code with Checkly Once you’ve gathered code changes into a pull request, and the updates go on from your local or deve environments, it’s time to do some further phases of the testing pyramid, namely integration testing, contract testing, and e2e testing. With tests running on Checkly you get: * Tests run from real locations with full geographic coverage * Consistent runtime environments - the same containers that run your Playwright tests against production will be used for staging tests * Easily accessible results - use the Checkly dashboards to view test results and traces In this section we’ll cover the necessary tools for running your code in Staging, with the appropriate configuration before it moves to production. ### Monitoring as Code This guide can’t provide a complete tutorial of the monitoring as code (MaC) workflow, but in general this will look something like: * Write tests locally from a shared repository * Update and expand your tests via source control * Deploy from your shared repository to Checkly * Manage configuration via config file in your repository * integrate ad-hoc test runs within your CI/CD tools [A complete map to MaC is on our documentation site](https://www.checklyhq.com/guides/monitoring-as-code/), and with these tools you can easily share monitoring code and configuration with your own team. ### Local Environment Variables Local environment variables are used **at build time**. This means they are available when you run commands like `checkly test` or `checkly deploy`. These variables are tied to your local environment, whether it’s your laptop, or a CI/CD pipeline. **Example:** Imagine you have an email alert channel in your Checkly setup. You can define the email address using `process.env` in your code: ```jsx theme={null} const email = process.env.EMAIL || 'stefan@checklyhq.com'; ``` If you want to override this email address locally, you can set an environment variable in your terminal before running the Checkly CLI: ```bash Terminal theme={null} EMAIL=raccoon@checklyhq.com npx checkly deploy ``` When you deploy, Checkly will use the local variable (`raccoon@checklyhq.com`) instead of the default value. *** ### Remote Environment Variables Remote environment variables are used **at runtime** in the Checkly cloud. These variables are accessible when your checks (like browser checks or API checks) are running in Checkly’s infrastructure. Local variables won’t work here because the Checkly cloud doesn’t have access to your local environment. **Example:** Let’s say you have a browser check that visits a URL defined by a remote environment variable: ```jsx theme={null} const url = process.env.SCREENSHOT_URL || ''; ``` If `SCREENSHOT_URL` isn’t set remotely, the check will default to `checklyhq.com`. To set this variable in the Checkly cloud, you can use the Checkly CLI: ```bash Terminal theme={null} npx checkly env add SCREENSHOT_URL ``` Now, when the browser check runs in the Checkly cloud, it will use the updated URL (`https://docs.checklyhq.com`). *** **Overriding Remote Variables** Sometimes, you might want to override remote environment variables temporarily, like during a preview deployment. The Checkly CLI makes this easy with the `-e` flag: ```bash Terminal theme={null} npx checkly test -e SCREENSHOT_URL=https://google.com --record --verbose ``` This command overrides the `SCREENSHOT_URL` variable for that specific test run, allowing you to test different configurations without changing your remote settings permanently. ### Optional: Private locations for running Checkly within your cloud For enterprise users, you may want to run Checkly monitors from within your cloud. This is especially helpful if your API isn’t available on the public internet, or you want to run checks from a geographic location not covered by the Checkly system. If your development and staging locations are running on private clouds, a [Checkly Private Location](/docs/platform/private-locations/overview) makes testing easier. a diagram showing A container on your cloud running a Checkly Private Location

a diagram showing A container on your cloud running a Checkly Private Location

*A container on your cloud running a Checkly Private Location* Requirements: * A Checkly enterprise accounts * A container runtime (we test using Docker, but other runtimes should work) * Outbound internet access for the Agent to [https://agent.checklyhq.com](https://agent.checklyhq.com/) (proxy configuration is supported) * Access to your API or browser-based applications and services from the Private Location network Once you’ve set up your private location, pick your private location when creating checks. If no agents run for 10 minutes, the location turns off until agents reconnect. ### Checkly Traces for OpenTelemetry insights At any layer of running tests against your services, it’s helpful to connect the information from synthetic testing with backend trace data. With [Checkly Traces](/docs/resolve/traces/overview), it’s possible to export backend data captured with [OpenTelemetry](https://www.checklyhq.com/learn/opentelemetry/getting-started-with-observability/) and gather trace data from your backend services as they’re handling the requests sent during a Checkly Test. Implementation is straightforward and results in a highly efficient observability setup that doesn’t use bandwidth for unnecessary monitoring data. In most cases the setup will look like this: 1. Configure Checkly to send a header with its requests, which will be picked up and propagated by OpenTelemetry instrumentation 2. Add a configuration to the [OpenTelemetry collector](https://www.checklyhq.com/learn/opentelemetry/what-is-the-otel-collector/) to send only relevant traces to Checkly 3. Enjoy information on database requests, compute time, third party requests, and other backend processes right in your Checkly dashboard a dashboard showing back-end requests

*With Checkly Traces you can see the database calls and other timespans as your backend services handle requests kicked off by a Checkly test.* Checkly Traces is easy to implement even if you aren’t currently using OpenTelemetry, [check out our documentation](/docs/resolve/traces/overview) on how to get started. ## 3. Production - Finding failures in real time with Playwright & Checkly ### Moving to Production cadence Once you’ve moved to production, it’s time to run your tests on a cadence, creating a monitor that tell you within minutes if any part of your service goes down. The process for setting this cadence looks like: 1. [Determine the right check frequency](https://www.checklyhq.com/blog/check-frequency/) based on SLA 2. Configure your tests either in the web UI or by editing your [test repository’s config](/docs/constructs/project) file 3. [Group or tag tests](/docs/platform/groups) by criticality to have consistent cadence for high value services - this also makes it easier to see checks together in the web UI With alerts running on a cadence, you can ensure that you’re hearing about failures before your users report them. The final step is setting up the alerting your team receives when a monitor detects problems. ### Alerting on failures Finally, now that we’re monitoring production with Checkly, we’ll need to send alerts to an on-call team and/or key stakeholders when something is detected as not working. [Checkly supports multiple alert channels](/docs/communicate/alerts/overview), with configuration available either for a single check, at the group level, or all checks. Some best practices: * Configure retry logic for minimal alert fatigue - Checkly offers fine-grained retries on checks, and it’s important to double-check that failures aren’t transitory. Balance alert fatigue with time efficiency, as multiple retries can significantly impact time to detection. * Don’t rely on a single alert channel, always have at least one backup like a slack message or email to make sure that a delivery failure doesn't aggravate an outage. * Use soft assertions and degraded states for more depth - not every behavior counts as total failure. For slowly-loading graphics or error messages that don’t block the user, [use the degraded state](https://www.checklyhq.com/blog/never-miss-alerts/#use-soft-assertions-to-reduce-alert-fatigue) to flag the problem without sending high priority alerts. There’s also a general factor of time and team culture: if you do the work now to ensure checkly alerts are timely and informative, your team will grow a culture of taking alerts from Checkly seriously, and either resolving or investigating every high priority notification. ## Conclusions: from dev to production with Checkly With Checkly and Playwright, you can test and monitor every stage—from local development to production—using the same powerful tools. This unified approach catches issues early, keeps your team informed, and helps you deliver a better experience to users. Start small with local tests, expand to staging, then monitor production with confidence. Checkly makes it easy to stay ahead of problems before they reach your users. # How to Use Setup and Teardown Scripts for Better API Monitoring Source: https://www.checklyhq.com/docs/guides/setup-scripts-for-apis Setup and teardown scripts are fundamental tools to tailor API checks to your target endpoints. This guide presents real-world examples to help you master these tools. ## The importance of self-contained checks Checkly's [API Monitoring](/docs/learn/monitoring/api-monitoring) checks run independently and on different schedules, possibly even following different retry logic when failures occur. It is therefore important not to create dependencies between them, which could introduce false failures and flakiness. Checkly prevents this antipattern by having each check run fully isolated in its own sandbox. But how do you run complex API checks that have prerequisites (e.g., auth tokens, test data) that themselves require an API call or other preparation steps? That's what [setup and teardown scripts](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown) are made for: * **Setup scripts** run before the API check's main request. Use them to prepare test data, fetch tokens, or configure the request. * **Teardown scripts** run after the HTTP request completes. Use them to clean up test data, scrub sensitive information, or normalize responses. ## Setup scripts Setup scripts prepare everything needed before the main HTTP request runs. Common tasks include: * HTTP requests to other services (and parsing responses) * Encrypting data payloads * Generating unique IDs, timestamps, and date strings To pass data from a setup script into the main HTTP request, you have direct access to the `request` object: | Property | Description | Type | | ------------------------- | ----------------------------------------------- | ------ | | `request.method` | The HTTP request method, e.g., 'GET', 'POST' | String | | `request.url` | The request URL. Query parameters are appended. | String | | `request.body` | The request body in string format. | String | | `request.headers` | The request headers. | Object | | `request.queryParameters` | The request query parameters. | Object | For example, set the `Authorization` header with `request.headers['Authorization'] = my_token`, or set the request body with `request.body = JSON.stringify({ id: 123 })`. > Available libraries depend on which [Runtime](/docs/platform/runtimes/overview) you have chosen. See the [runtime specification](/docs/platform/runtimes/runtime-specification) for details. ### Prepare test data A self-contained check is responsible for preparing (and cleaning up) all the test data it needs. Let's look at an example. We [use Checkly to monitor Checkly](https://blog.checklyhq.com/how-we-monitor-checkly/)! One of our checks verifies that `DELETE /v1/checks/{id}` works correctly. To verify that, we need a check to actually delete. We can create a new check in the setup script, retrieve its unique `id`, and pass it to the main HTTP request. Our main request hits the `DELETE` endpoint at `https://api.checklyhq.com/v1/checks`: checkly API check http request config

The request includes the `Authorization` header set to `Bearer `. Now for the setup script. We use [axios](https://axios-http.com/) to create a dummy check: ```js SetupScript.js theme={null} const axios = require("axios") const apiKey = process.env.API_KEY const { data } = await axios({ method: "post", url: "https://api.checklyhq.com/v1/checks", headers: { Authorization: `Bearer ${apiKey}` }, data: { name: "dummy_check", checkType: "BROWSER", frequency: 5, activated: true, locations: ["us-east-1"], script: 'const assert = require("chai").assert;const puppeteer = require("puppeteer");const browser = await puppeteer.launch();const page =await browser.newPage();await page.goto("https://google.com/");const title = await page.title();assert.equal(title, "Google"); await new Promise(resolve => setTimeout(resolve, 1000)); browser.close();', useGlobalAlertSettings: true, degradedResponseTime: 10000, maxResponseTime: 20000, } }) const checkId = data.id request.url = request.url + "/" + checkId ``` We've built a fully autonomous API check that creates everything it needs to verify the delete endpoint works correctly. Note that this check might still fail due to an issue with the `POST /v1/checks` endpoint used in the setup script, not the `DELETE` endpoint we're testing. The Checkly check result indicates whether the error occurred in the setup script or in the main HTTP request. ### Fetching dynamic test data Sometimes you want to fetch test data from an external source—maybe the data changes often, or you want to make checks more dynamic. This example creates a new product for a webshop using dynamic data: 1. Use a setup script to request random device information from an external API. 2. Map that data to match the target API's schema. 3. Send the data to `fakestoreapi.com`. Our main request hits the `POST` endpoint at `https://fakestoreapi.com/products`: checkly API check http request

The setup script fetches random device information and maps it to the product schema: ```js SetupScript.js theme={null} const axios = require('axios') const { data } = await axios({ method: "GET", url: "https://random-data-api.com/api/device/random_device", }) const responseData = { title: data.model, price: data.id, description: data.manufacturer, category: "device" } request.body = JSON.stringify(responseData) ``` This creates an encapsulated check that fetches dynamic test data from an external API and uses it to test a webshop endpoint. ## Teardown scripts Teardown scripts run after the HTTP request completes but before assertions evaluate. They have access to both the `request` and `response` objects. Common use cases include: * Cleaning up test data created during the check * Scrubbing sensitive data from responses before logging * Normalizing responses for consistent assertions ### Cleaning up test data If your setup script creates test data, use a teardown script to clean it up. This keeps your test environment clean and prevents data accumulation. Building on the earlier example where we created a dummy check to test the delete endpoint, what if we're testing a `GET` endpoint instead? We'd need to clean up the dummy check after the test: ```js TeardownScript.js theme={null} const axios = require("axios") const apiKey = process.env.API_KEY // The setup script stored the check ID in an environment variable const checkId = process.env.DUMMY_CHECK_ID if (checkId) { await axios({ method: "delete", url: `https://api.checklyhq.com/v1/checks/${checkId}`, headers: { Authorization: `Bearer ${apiKey}` } }) console.log('Cleaned up dummy check:', checkId) } ``` ### Scrubbing sensitive data Before response data is logged or stored, you might need to redact sensitive information for compliance reasons: ```js TeardownScript.js theme={null} const body = JSON.parse(response.body) if (body.user) { body.user.ssn = '[REDACTED]' body.user.creditCard = '[REDACTED]' body.user.email = body.user.email.replace(/(.{2}).*(@.*)/, '$1***$2') } response.body = JSON.stringify(body) ``` ### Error handling differences Setup and teardown scripts have different error handling behavior: | Aspect | Setup Scripts | Teardown Scripts | | ------------ | ---------------------------- | ------------------- | | On error | Check **aborts** immediately | Check **continues** | | HTTP request | Never executes | Already completed | | Assertions | Never run | Still evaluate | This means teardown scripts are safe for cleanup operations—if they fail, your assertions still run and you'll know if the main request worked. ## Combined patterns ### Create-then-delete workflow For testing CRUD operations, you often need to create a resource, test operations on it, and then delete it: **Setup script:** ```js SetupScript.js theme={null} const axios = require("axios") // Create a test user const { data } = await axios.post('https://api.example.com/users', { name: 'Test User', email: `test-${Date.now()}@example.com` }, { headers: { Authorization: `Bearer ${process.env.API_KEY}` } }) // Store the ID for the main request and teardown process.env.TEST_USER_ID = data.id request.url = `${request.url}/${data.id}` ``` **Main request:** `GET https://api.example.com/users` (tests retrieving the user) **Teardown script:** ```js TeardownScript.js theme={null} const axios = require("axios") const userId = process.env.TEST_USER_ID if (userId) { await axios.delete(`https://api.example.com/users/${userId}`, { headers: { Authorization: `Bearer ${process.env.API_KEY}` } }) console.log('Deleted test user:', userId) } ``` ### Conditional cleanup Sometimes you only want to clean up if the main request succeeded: ```js TeardownScript.js theme={null} const axios = require("axios") // Only clean up if the request was successful if (response.statusCode >= 200 && response.statusCode < 300) { const body = JSON.parse(response.body) await axios.delete(`https://api.example.com/orders/${body.orderId}`, { headers: { Authorization: `Bearer ${process.env.API_KEY}` } }) console.log('Cleaned up test order') } ``` ## Read more

Full reference for setup and teardown scripts, including built-in variables and technical details. Understand monitoring as code (MaC) via our Checkly CLI. Learn end-to-end monitoring with puppeteer and playwright to test key website flows.

# A Step-by-Step Engineering Guide for Checkly Source: https://www.checklyhq.com/docs/guides/startup-guide-detect-communicate-resolve How to get started with Checkly and Monitoring as Code. If you work in Operations — whether in DevOps or as an SRE — you know that technical failures, from bugs to edge cases to full-blown outages, are the primary reason users lose faith in your service. The better your team handles issues, the better your service will meet user expectations and SLAs. Every issue requires excellent tools to *detect*, *communicate*, and *resolve* problems, and Checkly offers industry-leading solutions for all three. This guide is designed for DevOps engineers and SREs who work to resolve and prevent technical failures that impact the user experience. ## Why teams use Checkly With Checkly, you can identify failures before your users do, resolve issues faster, and communicate more effectively with both users and team members. Here's what makes Checkly different from other monitoring tools: * **Proactive Monitoring** - Checkly uses uptime and synthetic monitoring to actively measure and alert teams of downtimes and degraded performance in their web applications, APIs, or other services. * **Monitoring as Code Workflow** - Your entire monitoring process - from checks, to error thresholds, to alert sequencing - can all be configured and scripted using the libraries and languages your engineers use today. Changes to your monitoring can be tracked and controlled with source control, just like your application code. * **Native Open Source Support** - Checkly enables end-to-end and transaction monitoring by leveraging Playwright, OpenAPI, Terraform, or Pulumi to run automated monitors globally, in production. Teams from [LinkedIn](https://www.checklyhq.com/case-study/modernizing-linkedins-monitoring-infrastructure/) to [Render](https://www.checklyhq.com/case-study/render/) have modernized their infrastructure and cut their debugging time with Checkly. ## The Checkly Tutorial: Build Faster With Monitoring as Code For this tutorial, we'll monitor the status of multiple pages, with the ability to simulate deep interaction with your pages through an automated browser. Throughout, we'll follow Checkly's [Monitoring as Code](https://www.checklyhq.com/learn/monitoring/monitoring-as-code/) approach, setting up everything through the CLI and your preferred IDE. The result will be a Checkly dashboard that offers a high level view or your services' status, and deep insight into the causes of failures. Here's a preview of our goal:

*Once you complete this tutorial, you'll have an easy way to monitor your service from locations across the globe, share check results with your team, and resolve issues faster.* We'll walk you through every step of the process—all you need is a site, API, or online service to monitor. You may have already tried adding monitors from the Checkly web UI. While you can configure everything through the web UI, Monitoring as Code is more comfortable for engineers and enables faster setup of comprehensive monitoring. A huge benefit of monitoring as code is managing your monitors with source control, meaning any update to your monitoring configuration is trackable. Never again will you wonder if something changed about a monitor that is suddenly failing, you'll know how it changed, when it was updated, and by whom! ## Step 1: Detect Problems This section will guide you through: 1. Setting up the Checkly CLI and your monitoring repository 2. Creating uptime monitors with URL checks 3. Building synthetic monitoring with API monitoring and Playwright browser checks (no prior Playwright experience needed) ### Set up the Checkly CLI and your project Start by creating a new project with: ```bash theme={null} npx checkly init ``` You’ll get a few quick prompts on the name and setup of your project: ```bash theme={null} Hi Nica Mellifera! Let's get you started on your monitoring as code journey! ✔ Where do you want to create your new project? … detect-comm-resolve ◼ Cool. Your project will be created in the directory "/Users/nica/checkly/examples/detect-comm-resolve" ? Which template would you like to use for your new project? › - Use arrow-keys. Return to submit. ❯ An advanced TypeScript project with multiple examples and best practices (recommended) An advanced JavaScript project with multiple examples and best practices A boilerplate TypeScript project with basic config A boilerplate JavaScript project with basic config ✔ Would you like to install NPM dependencies? (recommended) … yes ✔ Packages installed successfully ✔ Would you like to initialize a new git repo? (optional) … no ``` *Dialog on setting up a new Checkly repository.* This will create a new project with this structure: ```bash theme={null} __checks__/ # All monitoring checks ├── api.check.ts # API endpoint checks ├── heartbeat.check.ts # Heartbeat monitoring (commented out) ├── homepage.spec.ts # Browser-based Playwright tests └── url.check.ts # Simple URL monitoring package.json # Dependencies checkly.config.ts # Project configuration file ``` With a project set up, we can start writing our first monitors. Monitors come in two categories: simple and scalable [**uptime monitoring**](https://www.checklyhq.com/blog/announcing-checkly-uptime-monitors-simple-scalable/), and the more complex [**synthetic monitoring**](https://www.checklyhq.com/learn/monitoring/synthetic-monitoring/). ### Adding Uptime Monitoring for Every Endpoint All monitors are created as `.check.ts` files implementing the [Checkly CLI construct](/docs/constructs/overview). Here’s an example URL Monitor: ```ts theme={null} import { Frequency, UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs' // Import required constructs from Checkly CLI new UrlMonitor('url-monitor-example', { // Create new URL monitor with unique logical ID name: 'Example URL Monitor', // Human-readable name for the monitor activated: true, // Enable the monitor to run checks frequency: Frequency.EVERY_1M, maxResponseTime: 10000, // Maximum response time in milliseconds before marking as failed degradedResponseTime: 5000, // Response time in milliseconds for degraded performance warning request: { // Configure the HTTP request settings (required) url: 'https://httpbin.org/get', // Target URL to monitor followRedirects: false, // Do not follow HTTP redirects (3xx responses) skipSSL: false, // Validate SSL certificates (recommended for security) assertions: [ // Define validation rules for the response UrlAssertionBuilder.statusCode().equals(200), ] } }) ``` To create a new URL monitor, add a new file like `url-monitor.check.ts` in the `__checks__/` directory. Import `UrlMonitor` and `UrlAssertionBuilder` from `checkly/constructs`, then instantiate a new monitor with a unique logical ID, name, and request configuration. The monitor will send GET requests to your specified URL and validate HTTP status codes with `UrlAssertionBuilder.statusCode().equals(200)` to ensure the endpoint returns successful responses. URL monitors have limited assertion features, they’re primarily there to guarantee uptime with a broad reach. Uptime monitoring also includes low level checks like [TCP monitors](/docs/detect/uptime-monitoring/tcp-monitors/overview), and [Heartbeat monitors](/docs/detect/uptime-monitoring/heartbeat-monitors/overview) which listen for pings from your automated tasks. For deeper synthetics monitoring of your endpoints, to test and monitor correctness of responses in depth, reach for synthetics checks like API checks. Thankfully, the constructs signature is very similar: ```ts theme={null} import { ApiCheck, AssertionBuilder } from 'checkly/constructs' new ApiCheck('books-api-advanced-check', { name: 'Books API - Advanced Validation', alertChannels: [], //we'll add alert channes below. Currently this check will not send alerts! degradedResponseTime: 5000, maxResponseTime: 15000, request: { url: 'https://danube-web.shop/api/books', method: 'GET', followRedirects: true, skipSSL: false, headers: [{ key: 'X-My-Header', value: 'My custom header value' }], assertions: [ // Validate HTTP response status AssertionBuilder.statusCode().equals(200), // Check response headers AssertionBuilder.headers('content-type').contains('application/json'), // Validate first book has required fields AssertionBuilder.jsonBody('$[0].id').isNotNull(), AssertionBuilder.jsonBody('$[0].price').isNotNull(), // Check that title and author are not empty AssertionBuilder.jsonBody('$[0].title').notEmpty(), AssertionBuilder.jsonBody('$[0].author').notEmpty(), // Validate price is positive AssertionBuilder.jsonBody('$[0].price').greaterThan(0), // Check that we have multiple books in the catalog AssertionBuilder.jsonBody('$.length').greaterThan(1), // Validate response time AssertionBuilder.responseTime().lessThan(3000) ], }, runParallel: true, }) ``` All of these assertions are available in the `AssertionBuilder` [section of the constructs API](/docs/constructs/overview#assertionbuilder). You can go even further with API checks: from running scripts before each check to evaluating JSON responses. These advanced options are [covered in detail in our documentation site](/docs/constructs/overview#apicheck). With Checkly CLI constructs you can create and configure every type of monitor, and feed it any variation of configuration. Some extensions of API checks include: * Using [variables and secrets](/docs/platform/variables) to configure checks at runtime. Environment variables can be scoped to your whole account, or specific to a group of checks. * Execute arbitrary code (e.g. to fetch and assign session tokens), and clean up after yourself (e.g. to remove any new records created as part of the check), with [setup and teardown scripts for API checks](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown). ### Create and Manage Multiple Monitors at Once We can extend the principles here to create multiple monitors from a single file. In this example we provide an array of URLs to monitor and create as many URL monitors as we need. ```ts theme={null} import { UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs' // Configuration: URLs to monitor const urlsToMonitor = [ 'https://danube-web.shop/', 'https://danube-web.shop/api/books', 'https://httpbin.org/get', 'https://httpbin.org/status/200' ] // Loop through each URL and create a monitor urlsToMonitor.forEach((url, index) => { // Extract domain name for monitor ID const domain = new URL(url).hostname.replace('www.', '') // Generate unique logical ID for each monitor const logicalId = `multi-url-monitor-${index + domain}` new UrlMonitor(logicalId, { name: `Multi URL Monitor - ${url}`, activated: true, request: { url: url, followRedirects: true, skipSSL: false, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) }) ``` *This example shows the power of constructs for creating monitors. However, real world implementations would probably query a list of URLs from an endpoint rather than having the array of URLs hard-coded.* > Notice that the check above isn’t configured with a `maxResponseTime` , `alertChannels` , and several other settings. Without these settings the project defaults in `checkly.config.ts` will be used. We could also assign checks to a group to manage a subset of checks with one configuration. [More detail on groups is covered in the Checkly documentation](/docs/platform/groups). There’s more detail on alert thresholds and how alerts are communicated in part 2, below. We can test this file (and the others added above) with the `npx checkly test` command, which will scan the `/__checks__` directory and run all of our checks through the Checkly monitoring network. ```bash theme={null} __checks__/multiple-url-monitor.check.ts ✔ Multi URL Monitor - https://danube-web.shop/ (223ms) ✔ Multi URL Monitor - https://danube-web.shop/api/books (222ms) ✔ Multi URL Monitor - https://httpbin.org/get (301ms) ✔ Multi URL Monitor - https://httpbin.org/status/200 (284ms) __checks__/url-monitor-example.check.ts ✔ Example URL Monitor (2s) __checks__/books-api-advanced-check.check.ts ✔ Books URL (233ms) 6 passed, 6 total ``` With tests passing, we’re ready to deploy our first set of checks. * Open `checkly.config.ts` and edit the `projectName` and `logicalId` values in the config file. You’ll also want to look at the other defaults set here to make sure they make sense as global defaults for your Checkly project. * Run `npx checkly deploy -p` to see a preview of what checks will be created, make sure this list is consistent with what you expect to change in your Checkly project. * Now you’re ready to run your `npx checkly deploy` command. Here’s the response from a deployment: ```bash theme={null} $npx checkly deploy Parsing your project... ✅ Validating project resources... ✅ Bundling project resources... ✅ ✔ You are about to deploy your project "New Walkthrough Project" to account "nica@checklyhq.com". Do you want to continue? … yes Successfully deployed project "New Walkthrough Project" to account "nica@checklyhq.com". ``` ### Create Synthetic Monitors with Playwright While uptime monitors will ensure that our services are responding and their responses contain valid-looking data, to really test every edge case and user path through our application, we want synthetic monitoring. With [Playwright](https://www.checklyhq.com/learn/playwright/what-is-playwright/), we can simulate a user’s behavior in a browser, and get consistent, traceable results back from our checks. If you haven’t written any Playwright tests before, don’t worry. This tutorial will show you how to go from simple button clicking to simulating complex user behavior without having to learn a whole new framework. To create our first Playwright check, we’ll use the Playwright `codegen` tool to capture our behavior in a browser: ```bash theme={null} npx playwright codegen https://[your site].com ``` A Chromium browser and a new "Playwright inspector" window should appear. As you click through your site, you’ll see `codegen` generating the test script that will follow your path. the codegen playwright inspector

*`codegen` captures your inputs and lets you add assertions, the process is [fully documented on our documentation site](https://www.checklyhq.com/learn/playwright/codegen/).* Once you’re happy with the scripted test, copy the code into a `.spec.ts` file, and create a construct for your check as a `.check.ts` file, based on the CLI [constructs; this time for browser checks](/docs/constructs/overview#browsercheck). Here’s an example of a check for the demo site [Danube Web Shop](https://danube-web.shop/): ```ts {title="homepage-browse.check.ts"} theme={null} import { Frequency, AlertEscalationBuilder, BrowserCheck, RetryStrategyBuilder } from 'checkly/constructs' new BrowserCheck('browse-danube-homepage-v1', { name: 'Browse Danube Homepage', code: { //where to look for the Playwright script for this check entrypoint: './homepage-browse.spec.ts', }, //the settings below are common to all check types mentioned above activated: true, muted: true, shouldFail: false, locations: [ 'us-east-1', 'us-west-1', ], frequency: Frequency.EVERY_30M, // what must happen to generate an alert alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(1, { amount: 0, interval: 5, }, { enabled: false, percentage: 10, }), //how are retries spaced, and how many to perform retryStrategy: RetryStrategyBuilder.linearStrategy({ baseBackoffSeconds: 60, maxRetries: 2, maxDurationSeconds: 600, sameRegion: true, }), runParallel: true, }) ``` ```ts {title="homepage-browse.spec.ts"} theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { // Navigate to the homepage await page.goto('https://danube-web.shop/'); // Click on text element containing "Haben oder haben" await page.getByText('Haben oder haben').click(); // Click the "Add to cart" button to add the item to shopping cart await page.getByRole('button', { name: 'Add to cart' }).click(); // Click on a generic link element (breadcrumb) await page.getByRole('link').click(); // Click on a link containing "Fantasy" text (a category filter) await page.locator('a').filter({ hasText: 'Fantasy' }).click(); // Assert that "The Polar Turtle" product text is visible on the page await expect(page.getByText('The Polar Turtle')).toBeVisible(); // Assert that the price "$9" is visible (using .first() to get the first occurrence) await expect(page.getByText('$9').first()).toBeVisible(); }); ``` Much of the content of the `.check.ts` file will look familiar by this point, as much of this configuration is shared by all check types. If you’ve not used Playwright tests before, all of `homepage-browse.spec.ts` will be new. Playwright’s `codegen` tool will generate best practices Playwright, using the preferred type of locators for each element you click. ### Playwright Best Practices You can see in the script one of the many advantages of the Playwright framework by noticing what’s missing: there are no `wait()` lines for the check to pause and wait for the site to load! [Playwright’s auto-waiting](https://www.checklyhq.com/blog/playwright-auto-waiting/) makes it unnecessary to add manual waiting to have our script work. Instead, Playwright will continually attempt to click the item, and do so as soon as it’s available. In this first check, we’ve made only straightforward assertions of visible page elements, but this script will run in a full environment when deployed to Checkly, so we can do any type of scripted assertion we like: compare the page contents to the result of an API request, or parse the page in detail to compare to configuration value, it’s all possible. Our documentation site goes very [deep into Playwright fundamentals and advanced use cases](https://www.checklyhq.com/learn/playwright/what-is-playwright/). Once our check looks right, it’s time to run ```bash theme={null} npx checkly test homepage-browse ``` The same command used in a previous section ran everything in the `/__checks__` directory, but we can specify a file or part of a name to only run matching tests. ```bash theme={null} Running 1 checks in eu-west-1. __checks__/homepage-browse.check.ts ✔ Browse Danube Homepage (6s) 1 passed, 1 total ``` ## Part 2. Communicate Outages With Your Team Now that issues have been detected with proactive monitoring, we need to plan how our team will find out about problems. Checkly lets you customize what notifications your team receives, the information in those notifications, and the thresholds where a monitor decides that a particular check is failing. ### Configure When Alerts Go Out By setting up alert rules, you can define conditions—such as consecutive failures or specific error responses—that trigger alerts. You can also configure how to handle retries before generating an alert, and whether reminder messages go out for an ongoing alert. This is particularly useful for handling transient network issues or brief API hiccups. Well-tuned alerting policies allow your DevOps or SRE team to quickly identify and investigate issues before they escalate, without burdening the team with many false alarms. All of these settings are covered by the `alertEscalationPolicy` property of any monitor construct. Build it with [`AlertEscalationBuilder`](/docs/constructs/alert-escalation-policy) rather than writing the object by hand: ```ts theme={null} import { AlertEscalationBuilder } from 'checkly/constructs' // Escalate after 1 consecutive failed run; send no reminders alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(1, { amount: 0, // Number of reminder notifications to send (0 = no reminders) interval: 5, // Minutes between reminder notifications }) // Or escalate once the check has been failing for 5 minutes alertEscalationPolicy: AlertEscalationBuilder.timeBasedEscalation(5) ``` These settings just like all the settings in this section can be managed at the individual check level, [as a group](/docs/platform/groups), or across the whole project in the `checkly.config.ts` file. ### Manage alert channels Along with carefully controlling when alerts go out after detecting failure, it’s important to control alert channels. To ensure your team is promptly notified of outages, configure Checkly alerts to send notifications through channels like email, Slack, PagerDuty, or webhooks. In your Checkly CLI constructs, you can create alert channels like this example which creates an SMS alert channel: ```ts {title="sms-alert-channel.ts"} theme={null} import { SmsAlertChannel } from 'checkly/constructs' const smsChannel = new SmsAlertChannel('sms-channel-1', { name: 'Ops on-call', phoneNumber: '+31061234567890', }) ``` Here's another alert channel, this one for Slack using the Checkly Slack App. ```ts {title="slack-app-alert-channel.ts"} theme={null} import { SlackAppAlertChannel } from 'checkly/constructs' export const slackChannel = new SlackAppAlertChannel('slack-channel-1', { slackChannels: ['#alerts'], sendRecovery: true, sendFailure: true, sslExpiry: true, }) ``` Now we can go back to our URL monitor from part 1 and update it to include multiple alert channels: ```ts {tile="url-monitor.check.ts"} theme={null} import { Frequency, UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs' import { emailChannel } from './email-alert-channel' import { slackChannel } from './slack-alert-channel' import { webhookChannel } from './webhook-alert-channel' new UrlMonitor('url-monitor-example', { name: 'Example URL Monitor', activated: true, frequency: Frequency.EVERY_1M, maxResponseTime: 10000, degradedResponseTime: 5000, alertChannels: [emailChannel, slackChannel, webhookChannel], // Array of alert channels to notify when check fails or recovers request: { url: 'https://httpbin.org/get', followRedirects: false, skipSSL: false, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) ``` Integrating alerting with incident management tools ensures seamless communication during outages. For example, connecting Checkly with [PagerDuty](/docs/constructs/overview#pagerdutyalertchannel) can [automatically open incidents](/docs/integrations/incident-management/pagerduty), while Slack alerts keep the team informed in real time. Of course you can also [send alerts out to a webhook](/docs/constructs/overview#webhookalertchannel) to connect with your DIY alerting solution. By combining proactive alerting with smart retry logic, you create a robust monitoring system that keeps your DevOps or SRE team focused on resolving critical issues. For communicating directly with your users, teams should [explore status pages](/docs/communicate/status-pages/overview) to automatically communicate downtime or degraded performance. ## Part 3. Resolve Issues Faster with Checkly Dashboards Now that the problem is detected and our team has been alerted, it’s time to repair the issue. This will take us away from Monitoring as Code and into the Checkly dashboard UI where we can get a high level view of monitor performance, then zoom in to individual failures and their causes. a Checkly dashboard

*From a high-level view it’s easy to see this API, while usually available, has repeated failures in the recent past.* For API checks, failed check runs will generate a report that contains the response received and the timing of the request.

*The detail view when an API check fails* For browser checks, Checkly provides [detailed traces and execution logs](https://www.checklyhq.com/guides/reading-traces/) that help you diagnose why a check failed, offering insights into issues like slow response times, HTTP errors, or script failures. By reviewing the **Trace View**, you can examine each step of a check’s execution, including network requests, console logs, and screenshots from the browser check’s run. This granular data allows your team to pinpoint whether a failure stems from backend latency, third-party API issues, or frontend rendering problems. Additionally, Checkly’s error messages and stack traces simplify debugging, helping your team quickly identify root causes and implement fixes. Whether the issue is a timeout, a broken assertion, or a resource loading failure, Checkly’s trace data ensures you have the context needed to resolve incidents efficiently.

*When your browser check fails, Checkly stores a Playwright trace to offer in-depth, moment-by-moment reviews of what the browser saw as it tried to load the page.* If you're interested in adding back-end tracing information, explore Checkly Traces to send back-end OpenTelemetry trace data to Checkly for requests initiated by Checkly synthetic requests. [See the whole process on our documentation site](/docs/resolve/traces/overview). When your browser check fails, Checkly stores a Playwright trace to offer in-depth, moment-by-moment reviews of what the browser saw as it tried to load the page.

*A failed browser check. With Checkly Traces we also get backend trace information from services like Redis and our Postgres database* ## Conclusion: Detect, Communicate, Resolve Technical failures are inevitable, but with Checkly, your team can minimize their impact by quickly detecting issues, communicating them effectively, and resolving them before users are affected. **Detect Problems Before Users Do** Checkly’s monitoring tools—from simple uptime checks to complex Playwright-based synthetic tests—ensure you catch issues early. By implementing **Monitoring as Code**, you maintain consistency, version control, and scalability across your monitoring setup. **Communicate Clearly During Outages** With customizable alerting policies and multiple notification channels (Slack, PagerDuty, SMS, and more), your team stays informed the moment an issue arises. Well-tuned alerting reduces noise while ensuring critical failures get immediate attention. Teams can use [status pages](/docs/communicate/status-pages/overview) to communicate directly with their users. **Resolve Issues Faster with Deep Insights** All checks offer high-level views of the status of your monitors, which can show patterns of performance over time. For API checks, you can see failed check’s full responses and the timing of the requests. Browser checks offer the most insight into failures with detailed Playwright traces, screenshots, and logging from the check execution. ### Further Reading * [Learn: Playwright](https://www.checklyhq.com/learn/playwright/) - Go deep on the best web automation framework. * [Use Checkly With an AI IDE](/docs/ai/overview) - For writing your monitoring faster and more consistently. * [Environment Variables](/docs/cli/environment-variables) - To configure checks at runtime. * [Use Setup Scripts for Better API Monitoring](https://www.checklyhq.com/guides/setup-scripts-for-apis/) - Advanced techniques for keeping monitoring code DRY. * [Monitoring as Code with Terraform & Pulumi](https://www.checklyhq.com/docs/cli/cli-vs-terraform-pulumi/) - The Checkly CLI isn't the only way to do Monitoring as Code. With Checkly, you don’t just monitor your services—you ensure they meet user expectations. Start detecting, communicating, and resolving issues faster today. 🚀 [**Try Checkly now**](https://www.checklyhq.com/) and build a more reliable service. # Use Uptime Monitoring as Your First Line of Defense Source: https://www.checklyhq.com/docs/guides/uptime-monitoring The three main components of uptime monitors. Uptime Monitoring is part of a ‘[reliability in depth](https://www.checklyhq.com/blog/the-defense-in-depth-approach-to-app-monitoring/)’ strategy of guaranteeing that your services maintain available and performant while allowing you to meet your Service Level Agreements (SLAs) with your customers. You can combine Uptime Monitoring with [Synthetinc Monitoring](https://www.checklyhq.com/learn/monitoring/synthetic-monitoring/) and full-stack traces for a deep, 360 degree view into your applications health. The following guide will focus on your first layer of defense, Uptime Monitoring, and show you how to monitor every endpoint, service, and URL that your service supports. As an extension, you may consider monitoring third-party services that your service relies on. This guide will cover the three types of uptime monitors we have at Checkly: * URL Monitors * TCP Monitors * Heartbeat Monitors ## URL Monitors While Synthetic Monitoring can simulate complex user behaviors in a browser and alert you if performance or functionality degrades, Uptime Monitoring excels at continous validation that URLs and Services are available. URL monitoring sends high-frequency requests and can alert you of failures within seconds. This can be useful for monitoring your homepage, landing pages, product pages, and even blog and docs. Our scalable, affordable approach of implementing monitors as code makes Checkly perfect for applying blanketed monitor coverage across your entire digitial footprint. It’s easy to create a URL Monitor from your IDE with Checkly CLI, just create a check file in your project with the `URLMonitor` type: ```tsx UptimeMonitoring.check.ts theme={null} //urlPinger.check.js import { Frequency, UrlMonitor, UrlAssertionBuilder } from 'checkly/constructs' new UrlMonitor('url-pinger-1', { frequency: Frequency.EVERY_1M, name: 'URL pinger 1', activated: true, request: { url: 'https://httpbin.org/get', skipSSL: false, followRedirects: true, assertions: [ UrlAssertionBuilder.statusCode().equals(200), ] } }) ``` *Note that this check is set to run every 60 seconds!* And deploy your check with the `npx checkly deploy` command. Along with monitoring your own services, both URL monitors and synthetics checks can monitor third party services. With the high frequency and low cost of URL monitors, they’re an easy way to ensure the services you rely on are up and running. ## TCP requests for TCP services While URL checks will send regular HTTP requests, we all still have services that will only respond to TCP requests, for example mail services and ftp services. For these there’s TCP checks. Just like the other checks in this guide, you can create monitors from your IDE with Monitoring as code. Here’s an example: ```tsx UptimeMonitoring.check.ts theme={null} import { TcpMonitor, TcpAssertionBuilder } from 'checkly/constructs' new TcpMonitor('hello-tcp-1', { name: 'Hello TCP', activated: true, maxResponseTime: 5000, degradedResponseTime: 4000, request: { hostname: 'tcpbin.com', port: 4242, data: 'ping\n', ipFamily: 'IPv6', assertions: [ TcpAssertionBuilder.responseTime().lessThan(1000), TcpAssertionBuilder.responseData().contains('ping') ] } }) ``` *A basic URL monitor ready to deploy* TCP monitors offer similar options to URL monitors, with the addition of assertions to evaluate the responses. Read more about [TCP check assertions on our docs site](/docs/constructs/tcp-monitor). A TCP check dashboard

*Results for TCP checks include the status of your SSL certification, and can send alerts if your certs are close to expiring.* TCP checks are also helpful at identifying any DNS issues, as these request will also perform DNS lookup if you provide a hostname and, like the other checks in this guide, [the results for a TCP check](/docs/concepts/results) will include DNS lookup time. ## The Heartbeat Monitor, for services that don’t accept incoming requests Another type of service you’ll want to monitor is one that doesn’t accept any kind of incoming requests. There’s really not much to getting one to work, just set up a heartbeat in the Checkly UI or with the Checkly CLI. For example, the following code ```tsx UptimeMonitoring.check.ts theme={null} //heartbeat.check.ts import { HeartbeatMonitor } from 'checkly/constructs' new HeartbeatMonitor('heartbeat-check-1', { name: 'Send weekly newsletter job', period: 7, periodUnit: 'days', grace: 2, graceUnit: 'hours', }) ``` will create a new Heartbeat monitor called ‘Send Weekly Newsletter job’ that expects a ping every 7 days, and will alert if 7 days and 2 hours pass without a request. After saving `hearbeatDemo.check.ts` to your project file, you can create this check with `npx checkly deploy` The Checkly deploy command returning to the CLI

The Checkly deploy command returning to the CLI

*The deploy command will return the URL for your heartbeat* The heartbeat monitor expects a HTTP request (GET or POST) to the ping URL once per period. Configure your monitored task to ping the URL when successful. Even this simplest of monitors still has a helpful dashboard of results. A Heartbeat check dashboard

### Conclusion Uptime monitoring is a critical component of maintaining reliable services and meeting SLAs, and Checkly provides versatile tools to ensure your systems—and the third-party services they depend on—remain operational. By integrating these monitors into your workflow—either through the Checkly CLI or UI—you can proactively detect and resolve issues before they impact users. With URL monitors for simple HTTP checks, TCP monitors for specialized services, and heartbeat monitors for background jobs, you’ll have full visibility into your system’s health. Start implementing these uptime monitoring strategies today to strengthen your "[defense in depth](https://www.checklyhq.com/blog/the-defense-in-depth-approach-to-app-monitoring/)" approach and keep your services running smoothly. For more details on configuring checks, explore [Checkly’s documentation](https://www.checklyhq.com/docs/) and take control of your reliability monitoring. # Send Alerts to Discord Source: https://www.checklyhq.com/docs/integrations/alerts/discord Configure Discord integration to receive real-time alerts from Checkly monitors Checkly integrates with [Discord](https://discord.com/) and can deliver failure, degradation, and recovery messages to any project in your Discord server / channel. 1. First, create a **Webhook**. Log in to Discord and go to "Server Settings" > "Integrations" and click "Create Webhook". setup checkly discord step 1

2. **Choose a name** for your integration like "Checkly" and [add this Checkly icon](https://cdn.checklyhq.com/logos/fat_racoon_square.png). Click the "Copy the Webhook URL" to...well...copy the Webhook URL setup checkly discord integration step 2

setup checkly discord integration step 2

3. Log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find Discord on the list, and click "Add channel". setup checkly discord integration step 3

setup checkly discord integration step 3

4. Give the alert channel a name and **paste the Webhook URL** in their respective input fields. You can now also tweak which alerts you want to be notified of and which checks or check groups should be subscribed to this channel. setup checkly discord integration step 4

setup checkly discord integration step 4

Note that we provide a preconfigured message payload but you are free to edit the payload and add more or different variables. Just click the "Edit payload" button and reference the "Help & variables tab". Congratulations! You have successfully integrated Checkly with Discord! ## Advanced Discord Configuration ### Custom Discord Alerts For more control over Discord alert formatting, you can create custom webhook payloads using Discord's embed format: ```json theme={null} { "embeds": [ { "title": "{{ALERT_TITLE}}", "description": "**Check**: {{CHECK_NAME}}\n**Location**: {{RUN_LOCATION}}\n**Response Time**: {{RESPONSE_TIME}}ms", "color": "{{#eq ALERT_TYPE 'ALERT_FAILURE'}}15158332{{else}}{{#eq ALERT_TYPE 'ALERT_DEGRADED'}}16776960{{else}}3066993{{/eq}}{{/eq}}", "fields": [ { "name": "Error", "value": "{{CHECK_ERROR_MESSAGE}}", "inline": false }, { "name": "Started At", "value": "{{STARTED_AT}}", "inline": true }, { "name": "Location", "value": "{{RUN_LOCATION}}", "inline": true } ], "footer": { "text": "Checkly Monitoring" }, "timestamp": "{{STARTED_AT}}", "url": "{{RESULT_LINK}}" } ] } ``` ### Discord Alert Colors Use different colors to distinguish alert types: * **Red (15158332)**: Failures and critical alerts * **Orange (16776960)**: Degraded performance alerts * **Green (3066993)**: Recovery notifications ### Discord Best Practices * Use separate channels for different environments (production, staging, development) * Configure appropriate notification settings to avoid alert fatigue * Use Discord's thread feature for grouping related alerts * Set up role mentions (@role) for critical alerts that require immediate attention # Email Notifications Source: https://www.checklyhq.com/docs/integrations/alerts/email Configure email alerts for individuals or distribution lists with customizable formatting and content **Monitoring as Code**: Learn more about the [Email Alert Channel Construct](/docs/constructs/email-alert-channel). Email notifications provide a reliable and widely-supported way to receive alerts from your Checkly monitors. Set up email alerts for individuals or distribution lists with customizable content. Light mode interface

## Basic Email Configuration To recieve alerts, you'll need to add a valid email address, choose what states you want to receive alerts for, and then also add subscriptions to checks. When an alert is triggered, you'll receive an email like below: mode interface

# Integrating GitLab Alerts Source: https://www.checklyhq.com/docs/integrations/alerts/gitlab_alerts Configure GitLab Alerts integration to receive failure, degradation, and recovery messages from Checkly Checkly integrates with [GitLab Alerts](https://docs.gitlab.com/ee/operations/incident_management/alerts.html) and can deliver failure, degradation, and recovery messages to any project in your GitLab installation. More specifically, Checkly will: * Open new alerts when a check fails. * Close alerts automatically when a failing check recovers. * Alert when SSL certificates are about to expire. 1. First, create an **HTTP Endpoint**. Log in to GitLab, select a project workspace and go to "Settings" > "Monitor" and expand the "Alerts" tab and click the "add new integration" button. Then select "HTTP Endpoint" as the integration type. 2. **Choose a name** for your integration like "Checkly". setup checkly gitlab integration step 2

3. Save the integration **copy the displayed Webhook URL and Authorization Key**. It should start with something like `https://gitlab.com/...` setup checkly gitlab integration step 3

4. Log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find GitLab Alerts on the list, and click "Add channel". setup checkly gitlab_integration step 5

5. Give the alert channel a name and **paste the Webhook URL and Authorization Key** in their respective input fields. You can now also tweak which alerts you want to be notified of and which checks or check groups should be subscribed to this channel. setup checkly gitlab integration step 5

> Note that we provide a preconfigured message payload but you are free to edit the payload and add more or different variables. Just click the "Edit payload" button and reference the "Help & variables tab". Congratulations! You have successfully integrated Checkly with GitLab Alerts! # Send Alerts to Microsoft Teams Source: https://www.checklyhq.com/docs/integrations/alerts/msteams Configure Microsoft Teams integration to receive failure, degradation, and recovery messages from Checkly **Monitoring as Code**: Learn more about the [MS Teams Alert Channel Construct](/docs/constructs/msteams-alert-channel). Checkly integrates with [Microsoft Teams](https://www.microsoft.com/en/microsoft-365/microsoft-teams/free) and can deliver failure, degradation, and recovery messages to any channel in any team. You can add as many Teams channels as you wish. To enable the MS Teams alert channel, take the following steps: 1. First, create a **Workflow**. Log in to your MS Teams account, go to Apps and search for *workflows*. setup checkly msteams_integration step 1

setup checkly msteams_integration step 1

2. Open the *Workflows* app, go to *Create* and select the **Post to a channel when a webhook request is received** template, name your workflow, and press "Next". setup checkly msteams integration step 2

setup checkly msteams integration step 2

3. Select which *Team* and *Channel* the integration should post alert notifications to, then press "Create flow". setup checkly msteams integration step 3

setup checkly msteams integration step 3

4. **Copy the displayed URL**. Then close the workflow setup by clicking "Done". setup checkly msteams integration step 4

5. Log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find MS Teams on the list, and click "Add channel". setup checkly msteams integration step 5

setup checkly msteams integration step 5

6. Give the alert channel a name and **paste the URL** in the dedicated URL input field. You can now also tweak which alerts you want to be notified of and which checks or check groups should be subscribed to this channel. setup checkly msteams integration step 6

setup checkly msteams integration step 6

7. Save your alert channel. The next time checks subscribed to this channel triggers an alert, the integration will post a message to the configured Channel. Below is an example of an alert message using the default configuration. checkly msteams integration example

> Note that we provide a preconfigured message payload but you are free to edit the payload and add more or different > variables. Just click the "Edit payload" button and reference the "Help & variables tab". Congratulations! You have successfully integrated Checkly with Microsoft Teams! # Phone Calls Source: https://www.checklyhq.com/docs/integrations/alerts/phone-calls Configure phone call notifications for critical alerts and emergency escalation **Monitoring as Code**: Learn more about the [Phone Call Alert Channel Construct](/docs/constructs/phone-call-alert-channel). The phone numbers used for phone call alerting need to be in [E.164 format](https://www.twilio.com/docs/glossary/what-e164). Stick to the following rules, and you'll be fine: * Prepend international access codes with a + sign * Do not use any white spaces * Use up to 15 characters send monitoring alerts with phone calls

You can add as many phone number channels as you like. ## Supported countries and regions | Country | | --------------------------- | | Argentina (+54) | | Australia (+61) | | Austria (+43) | | Belgium (+32) | | Brazil (+55) | | Canada (+1) | | Chile (+56) | | Czech Republic (+420) | | Denmark (+45) | | France (+33) | | Germany (+49) | | Hungary (+36) | | Iceland (+354) | | India (+91) | | Israel (+972) | | Italy (+39) | | Liechtenstein (+423) | | Lithuania (+370) | | Luxembourg (+352) | | Malta (+356) | | Mexico (+52) | | Netherlands (+31) | | Norway (+47) | | Paraguay (+595) | | Poland (+48) | | Portugal (+351) | | Spain (+34) | | Switzerland (+41) | | United Arab Emirates (+971) | | United Kingdom (+44) | | United States (+1) | | Uruguay (+598) | | Vietnam (+84) | If your country is not supported feel free to reach out to our [support team](mailto:support@checklyhq.com). ## Phone call number and sender ID Phone call alerts always use the following number: +18142508623. You can save the number as a contact to allow alerts bypassing any do-not-disturb settings. For limits and other phone call related questions, please refer to our [Pricing page](https://www.checklyhq.com/pricing/#features). # Slack Source: https://www.checklyhq.com/docs/integrations/alerts/slack Configure the Slack integration to receive real-time alerts from Checkly Setting up Slack as an alerting channel is simple: 1. First, install the Checkly Slack App into your Slack Workspace by going to [your integrations page](https://app.checklyhq.com/accounts/settings/account/integrations) and click "Connect Slack Workspace". 2. Then, head to [your alert settings](https://app.checklyhq.com/accounts/alerts/settings), click "Add more channels" and select a new Slack channel. Add a Slack WebHook URL to Checkly

You can add any **#public-channel-name**, **@username** for Direct Messages or a channel ID. For private channels, the Checkly Slack app needs to be explicitly invited to that channel by tagging it like `@Checkly`. If you're using a free Slack workspace, be aware of [message limits](https://slack.com/help/articles/115002422943-Usage-limits-for-free-workspaces). Exceeding these limits may prevent new alerts from being delivered. ## Example Failed Slack Alert For each check and / or monitor, the Slack alert message contains key information, a possible root cause analysis and key actions like rerunning the check. A slack message showing a failed alert

The root cause analysis uses AI to analyze your logs, traces, test source code, error information and other artefacts available to Checkly. Be aware that AI can make mistakes so outputs should be verified independently. ## Example Recovered Slack Alert From the recovered Slack message, you can see the timestamp as well as a link to the check itself. A slack message showing a recovered alert

A slack message showing a recovered alert

## Migrating from the deprecated Slack webhook alert channels Checkly previously only had Slack webhook based alert channels. These types of webhooks have been deprecated by Slack and are now also deprecated by Checkly. These webhook channels will keep working; we have not set a hard end-of-life date. You can recognize the deprecated channel in your alert channels list. It is named "Slack webhook" and has an alert icon. Deprecated Slack channel

To help you migrate from the old webhook based alert to the Slack app based alerts, we added a "Convert to Slack App alert channel" button in each current webhook-based alert channel. Just click the button, confirm which channels you want to migrate or add new ones, and click "Create Slack App alert channel".

After this, you can safely delete the deprecated channel. ## Permissions Checkly requests these Slack permissions for the Checkly Slack app to work within your workspace: | Permission | Description | | :------------------ | :------------------------------------------------------------------------------------------------------------- | | `app_mentions:read` | Detects @mentions to start Cloud Agents and respond to requests | | `channels:history` | Reads previous messages in threads for extra context | | `channels:join` | Automatically joins public channels when invited or requested. | | `channels:read` | Accesses channel metadata (IDs and names) to post replies and updates | | `chat:write` | Sends status updates and notifications on triggered actions like rerunning checks | | `groups:history` | Reads previous messages in private channels for extra context | | `groups:read` | Accesses private channel metadata to post responses | | `im:history` | Accesses direct message history for extra context | | `mpim:history` | Accesses group DM history for multi-participant conversations | | `users:read` | Matches Slack users with Checkly accounts for permissions and secure access | | `links:read` | Read links for allow-listed domains like [app.checklyhq.com](http://app.checklyhq.com) to provide rich context | | `links:write` | Writes rich context for allow-listed domains like [app.checklyhq.com](http://app.checklyhq.com) | ## Privacy Policy For information about how Checkly collects, uses, and protects your data, see our [Privacy Policy](https://checklyhq.com/privacy). # SMS Alerts Source: https://www.checklyhq.com/docs/integrations/alerts/sms Configure SMS text messages and phone call notifications for critical alerts and emergency escalation **Monitoring as Code**: Learn more about the [SMS Alert Channel Construct](/docs/constructs/sms-alert-channel). The phone numbers used for SMS alerting need to be in [E.164 format](https://www.twilio.com/docs/glossary/what-e164) format. Stick to the following rules and you'll be fine: * Prepend international access codes with a + sign * Do not use any white spaces * Use up to 15 characters send monitoring alerts with SMS

You can add as many SMS channels as you like. ## Supported countries and regions ### North America | | | | -------------------------------------------------------------------------------------- | -------------------------------- | | Anguilla (+1264) | Antigua and Barbuda (+1268) | | Aruba (+297) | Ascension (+247) | | Bahamas (+1242) | Barbados (+1246) | | Belize (+501) | Bermuda (+1441) | | Canada (+1) | Cayman Islands (+1345) | | Costa Rica (+506) | Cuba (+53) | | Curaçao and Caribbean Netherlands (Bonaire, Sint Eustatius, Sint Maarten, Saba) (+599) | Dominica (+1767) | | Dominican Republic (+1829) | Dominican Republic (+1849) | | Dominican Republic (+1809201) | Dominican Republic (+1809) | | El Salvador (+503) | Greenland (+299) | | Grenada (+1473) | Guadeloupe (+590) | | Guatemala (+502) | Haiti (+509) | | Honduras (+504) | Jamaica (+1876) | | Martinique (+596) | Mexico (+52) | | Montserrat (+1664) | Nicaragua (+505) | | Panama (+507) | Puerto Rico (+1787) | | St Kitts and Nevis (+1869) | St Lucia (+1758) | | St Pierre and Miquelon (+508) | St Vincent Grenadines (+1784) | | Trinidad and Tobago (+1868) | Turks and Caicos Islands (+1649) | | United States (+1) | Virgin Islands, British (+1284) | | Virgin Islands, U.S. (+1340) | | ### Asia | | | | ----------------------- | ---------------------------------- | | Afghanistan (+93) | Armenia (+374) | | Azerbaijan (+994) | Bahrain (+973) | | Bangladesh (+880) | Bhutan (+975) | | Brunei (+673) | Cambodia (+855) | | China (+86) | East Timor (+670) | | Georgia (+995) | Hong Kong (+852) | | India (+91) | Indonesia (+62) | | Iran (+98) | Iraq (+964) | | Israel (+972) | Japan (+81) | | Jordan (+962) | Korea Dem People's Rep (+850) | | Korea Republic of (+82) | Kuwait (+965) | | Kyrgyzstan (+996) | Laos PDR (+856) | | Lebanon (+961) | Macau (+853) | | Malaysia (+60) | Maldives (+960) | | Mongolia (+976) | Myanmar (+95) | | Nepal (+977) | Oman (+968) | | Pakistan (+92) | Palestinian Territory (+970, +972) | | Philippines (+63) | Qatar (+974) | | Saudi Arabia (+966) | Singapore (+65) | | Sri Lanka (+94) | Syria (+963) | | Taiwan (+886) | Tajikistan (+992) | | Thailand (+66) | Turkiye (+90) | | Turkmenistan (+993) | United Arab Emirates (+971) | | Uzbekistan (+998) | Vietnam (+84) | | Yemen (+967) | | ### Europe | | | | ---------------------------------- | ----------------------------------------- | | Albania (+355) | Andorra (+376) | | Austria (+43) | Belarus (+375) | | Belgium (+32) | Bosnia and Herzegovina (+387) | | Bulgaria (+359) | Canary Islands (+3491) | | Croatia (+385) | Cyprus (+357) | | Czech Republic (+420) | Denmark (+45) | | Estonia (+372) | Faroe Islands (+298) | | Finland/Aland Islands (+358) | France (+33) | | Germany (+49) | Gibraltar (+350) | | Greece (+30) | Guernsey/Jersey (+44) | | Hungary (+36) | Iceland (+354) | | Ireland (+353) | Isle of Man (+44) | | Italy (+39) | Kosovo (+383) | | Latvia (+371) | Liechtenstein (+423) | | Lithuania (+370) | Luxembourg (+352) | | Malta (+356) | Moldova (+373) | | Monaco (+377) | Montenegro (+382) | | Netherlands (+31) | Norway (+47) | | Poland (+48) | Portugal (+351) | | Republic of North Macedonia (+389) | Romania (+40) | | San Marino (+378) | Serbia (+381) | | Slovakia (+421) | Slovenia (+386) | | Spain (+34) | Sweden (+46) | | Switzerland (+41) | Turkish Republic of Northern Cyprus (+90) | | Ukraine (+380) | United Kingdom (+44) | | Vatican City (+379) | | ### South America | | | | ----------------------- | -------------------- | | Argentina (+54) | Bolivia (+591) | | Brazil (+55) | Chile (+56) | | Colombia (+57) | Ecuador (+593) | | Falkland Islands (+500) | French Guiana (+594) | | Guyana (+592) | Paraguay (+595) | | Peru (+51) | Suriname (+597) | | Uruguay (+598) | Venezuela (+58) | ### Africa | | | | ---------------------------- | ----------------------------- | | Algeria (+213) | Angola (+244) | | Benin (+229) | Botswana (+267) | | Burkina Faso (+226) | Burundi (+257) | | Cameroon (+237) | Cape Verde (+238) | | Central Africa (+236) | Chad (+235) | | Comoros (+269) | Congo (+242) | | Congo, Dem Rep (+243) | Djibouti (+253) | | Egypt (+20) | Equatorial Guinea (+240) | | Eritrea (+291) | Ethiopia (+251) | | Gabon (+241) | Gambia (+220) | | Ghana (+233) | Guinea (+224) | | Guinea-Bissau (+245) | Ivory Coast (+225) | | Kenya (+254) | Lesotho (+266) | | Liberia (+231) | Libya (+218) | | Madagascar (+261) | Malawi (+265) | | Mali (+223) | Mauritania (+222) | | Mauritius (+230) | Morocco/Western Sahara (+212) | | Mozambique (+258) | Namibia (+264) | | Niger (+227) | Nigeria (+234) | | Reunion/Mayotte (+262) | Rwanda (+250) | | Sao Tome and Principe (+239) | Senegal (+221) | | Seychelles (+248) | Sierra Leone (+232) | | Somalia (+252) | South Africa (+27) | | South Sudan (+211) | Sudan (+249) | | Swaziland (+268) | Tanzania (+255) | | Togo (+228) | Tunisia (+216) | | Uganda (+256) | Zambia (+260) | | Zimbabwe (+263) | | ### Oceania | | | | ----------------------- | -------------------------------------- | | American Samoa (+1684) | Australia/Cocos/Christmas Island (+61) | | Cook Islands (+682) | Fiji (+679) | | French Polynesia (+689) | Guam (+1671) | | Kiribati (+686) | Marshall Islands (+692) | | Micronesia (+691) | New Caledonia (+687) | | New Zealand (+64) | Niue (+683) | | Norfolk Island (+672) | Northern Mariana Islands (+1670) | | Palau (+680) | Papua New Guinea (+675) | | Samoa (+685) | Solomon Islands (+677) | | Tonga (+676) | Tuvalu (+688) | | Vanuatu (+678) | Wallis and Futuna (+681) | ## SMS alert number and sender ID SMS alerts will be sent with a sender ID and phone number. The Checkly sender ID is 'Checkly'. If your carrier does not support sender ID, you will instead see the Checkly SMS alert phone number, +1(814) 250 8623. For limits and other SMS-related questions, please refer to our [Pricing page](https://www.checklyhq.com/pricing/#features). # Send Alerts via Telegram Source: https://www.checklyhq.com/docs/integrations/alerts/telegram Learn how to send Checkly alerts to Telegram direct chats, groups, and forum topics Checkly integrates with [Telegram](https://telegram.org/) and can deliver failure, degradation, and recovery messages to direct chats, groups, and forum topics. You can add as many Telegram chats as you wish. * A Telegram bot and its HTTP API token * The Chat ID for the direct chat, group, or channel that should receive alerts * Optional: the Message Thread ID for a specific Telegram forum topic ## Telegram Bot 1. Create a **Telegram Bot** using the [@BotFather](https://t.me/botfather). Detailed instructions can be found [here](https://core.telegram.org/bots), however you can simply message the Telegram user `@BotFather` and send them the command message `/newbot`. Then follow the instructions the bot replies with. setup checkly telegram_bot step 1

2. As the last step of creating your new bot, the BotFather should reply with a message congratulating you that everything is complete and include a new HTTP API Token for you to use. Alternatively, if you want to reuse an existing Telegram bot you can message the BotFather with the command message `/token` and they will generate a new token. **Make sure to copy this token value out as we'll be using it in the Checkly setup later.** 3. Start your new Telegram bot by opening a chat with it. You can use the **START** button in the UI or type the `/start` command message. setup checkly telegram_bot step 2

## Chat ID For direct messages, you can retrieve your user Chat ID with an ID bot. For groups and forum topics, query the Telegram API to retrieve the target group's Chat ID and optional Message Thread ID. #### ID Bot You can retrieve your own Telegram Chat ID by starting a chat with another bot. This Telegram bot user can be found at `@get_id_bot`. 1. Send the `@get_id_bot` bot the `/my_id` command. Copy the ID from its reply. #### API Query Use the bot API to retrieve a Chat ID for any target chat. For a forum group, the same response includes the Message Thread ID, which Telegram also calls the topic ID. 1. Add your bot to the target chat or group. If you want alerts in a specific forum topic, send the bot a command from that topic. 2. In a browser, open `https://api.telegram.org/bot/getUpdates`, replacing `` with your bot's API token. setup checkly telegram_bot step 5

3. Find the message in the `result` array and copy these values: * `message.chat.id`: the Chat ID * `message.message_thread_id`: the optional Message Thread ID for the forum topic The relevant fields look like this: ```json Telegram update theme={null} { "message": { "message_thread_id": 42, "chat": { "id": -1001234567890, "type": "supergroup", "is_forum": true } } } ``` The `getUpdates` method does not work while your bot has an outgoing webhook configured. In that case, retrieve these fields from the update sent to your webhook. ## Checkly Integration With the **HTTP API Token**, **Chat ID**, and optional **Message Thread ID** in hand, you can create a Telegram alert channel in Checkly. 1. Log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find Telegram on the list, and click "Add channel". setup checkly telegram_bot step 6

2. Give the alert channel a name and paste the API Token and Chat ID into the dedicated fields. To send alerts to a specific forum topic, also paste its topic ID into **Message Thread ID**. You can then choose which alerts you want to receive and which checks or check groups should trigger this channel. Note that we provide a preconfigured message payload, but you may want to edit the payload to modify the Telegram alert message contents. Such as by adding more variables or changing existing variables. Click the "Edit payload" button and reference the "Help & variables" tab. Congratulations! You have successfully integrated Checkly with Telegram! ## Alternatives Telegram also accepts messages via an HTTP `GET` request. Therefore, you can use the API Token and Chat ID to create a custom Telegram alert channel with a [webhook alert channel](/docs/integrations/alerts/webhooks). The webhook should use the `GET` method and target `https://api.telegram.org/bot/sendMessage?chat_id=&text=Your%20Alert%20Title`. To target a forum topic, add `&message_thread_id=` before the `text` parameter. See Telegram's [`sendMessage` documentation](https://core.telegram.org/bots/api#sendmessage) for more information. ## Telegram Best Practices ### Group vs Direct Messages * **Direct Messages**: Use your personal chat ID for personal notifications * **Group Chats**: Add the bot to group chats and use the group's Chat ID for team notifications * **Forum Topics**: Use the group's Chat ID together with the topic's Message Thread ID * **Channels**: Bots can post to channels if added as administrators ### Message Formatting Telegram supports basic formatting in messages: ```text theme={null} *Check {{CHECK_NAME}} has failed* Location: {{RUN_LOCATION}} Response Time: {{RESPONSE_TIME}}ms Started: {{STARTED_AT}} Error: {{CHECK_ERROR_MESSAGE}} [View Details]({{RESULT_LINK}}) ``` ### Security Considerations * **Bot Token Security**: Keep your bot tokens secure and never share them publicly * **Chat ID Privacy**: Chat IDs can be sensitive information, especially for private groups * **Bot Permissions**: Only give your bot the minimum necessary permissions in group chats # Webhook Integrations Source: https://www.checklyhq.com/docs/integrations/alerts/webhooks Create custom webhook integrations to send Checkly alerts to any API endpoint **Monitoring as Code**: Learn more about the [Webhook Alert Channel Construct](/docs/constructs/webhook-alert-channel). Checkly Webhook Alert Channel

Webhooks allow you to POST custom payloads to any endpoint in your own infrastructure or a third party provider. In a nutshell, you can: * Create a **custom URL** by adding in authentication tokens or other secrets. * Create a **custom payload body** using any environment variables and specific instance variables per event. Note: that means that if you are attaching the webhook to a Group, you will be able to access Group-level variables, too. * **Debug and test the webhook** in the editor by sending test messages. ## Using Variables The example above shows a webhook configured to create a Jira ticket on each event. Notice the following: * We use the variable `JIRA_INSTANCE_URL` in the URL. We previously stored this variable in the [environment variables section](https://app.checklyhq.com/environment-variables). Alert-channel configurations accept both [environment variables](/docs/platform/variables) and [secrets](/docs/platform/secrets) using the same `{{NAME}}` syntax. * We use the variable `CHECK_ID` in the payload. This is one of many event-based variables that will change with each call. See below for the complete list. In both cases we use the familiar Handlebars templating braces, i.e. `{{ }}` to insert the variable. To avoid encoding, you can access your environment variables with triple brackets, i.e. `{{{USER_API_KEY}}}` **Using environment variables and secrets in this channel's config** Any literal-secret field in this alert channel — webhook URL, API key, bearer token, service key, etc. — accepts a [variable](/docs/platform/variables) or [secret](/docs/platform/secrets) reference using the familiar Handlebars syntax: `{{MY_VARIABLE}}`. You can also promote a value to a secret directly from the field: type the literal value, click the lock icon next to the input, and choose a name. The value is stored as an encrypted account secret and is never re-displayed in the UI. You can use the following event-related variables in both URL and payload. | Variable | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AI_ANALYSIS_CLASSIFICATION` | Classification of the failure / error. Available when Rocky AI automation is enabled. | | `AI_ANALYSIS_CODE_FIX` | A code fix to resolve the analyzed problem. Available when Rocky AI automation is enabled. | | `AI_ANALYSIS_LINK` | Direct link to full AI analysis. Available when Rocky AI automation is enabled. | | `AI_ANALYSIS_ROOT_CAUSE` | Description of possible root cause. Available when Rocky AI automation is enabled. | | `AI_ANALYSIS_USER_IMPACT` | Description of the user impact. Available when Rocky AI automation is enabled. | | `ALERT_TITLE` | Human readable title, e.g. 'Check "My API check" has failed' | | `ALERT_TYPE` | Type of alert, e.g. "ALERT\_FAILURE", "ALERT\_RECOVERY", "ALERT\_DEGRADED", "ALERT\_DEGRADED\_RECOVERY". See [alert states](/docs/communicate/alerts/overview#alert-states) for all options. | | `API_CHECK_RESPONSE_STATUS_CODE` | The response status code, e.g. 200. Only populated for API checks. | | `API_CHECK_RESPONSE_STATUS_TEXT` | The response status text, e.g. "OK". Only populated for API checks. | | `CHECK_ERROR_MESSAGE` | The check error message | | `CHECK_ID` | The UUID of the check | | `CHECK_DESCRIPTION` | The description of the check, if set | | `CHECK_NAME` | Full name of the check | | `CHECK_RESULT_ID` | The UUID of the result that triggered this message | | `CHECK_TYPE` | The check type, e.g. API, BROWSER . | | `GROUP_NAME` | The name of the group, if the check belongs to one. | | `IS_REMINDER` | Whether this alert is a reminder (`true`) or the initial alert (`false`). | | `REGION` | Region code where check ran, e.g. "us-east-1" | | `REMINDER_SEQUENCE` | The reminder number (`0` for initial alerts, `1`+ for reminders). | | `RESPONSE_TIME` | The reported response time for this result | | `RESULT_LINK` | The full link to the check result | | `RUN_LOCATION` | The location where the check ran, i.e. "N. California" | | `SSL_CHECK_DOMAIN` | The domain of the SSL certificate. For ALERT\_SSL only. | | `SSL_DAYS_REMAINING` | How many days remain on the SSL certificate. For ALERT\_SSL only. | | `STARTED_AT` | The ISO timestamp from when this check run started | | `TAGS` | An array of tags assigned to the check. Have a look at our Opsgenie example below on how to render this to a JSON array. | ### Using AI analysis data in alerts When [automatic Rocky AI analysis](/docs/resolve/ai-root-cause-analysis/overview#automatic-analysis) is enabled, you can include AI-generated root cause and user impact data in your webhook payloads. Use `{{#if}}` guards so the payload works whether AI analysis data is available or not. ```json theme={null} { "message": "{{ALERT_TITLE}}", "link": "{{RESULT_LINK}}", {{#if AI_ANALYSIS_ROOT_CAUSE}} "root_cause": "{{AI_ANALYSIS_ROOT_CAUSE}}", {{/if}} {{#if AI_ANALYSIS_USER_IMPACT}} "user_impact": "{{AI_ANALYSIS_USER_IMPACT}}", {{/if}} {{#if AI_ANALYSIS_LINK}} "ai_analysis_link": "{{AI_ANALYSIS_LINK}}", {{/if}} "classification": "{{AI_ANALYSIS_CLASSIFICATION}}" } ``` ### Handlebars Helpers Webhook templates support [Handlebars built-in helpers](https://handlebarsjs.com/guide/builtin-helpers.html), such as `if`, `unless`, and `each`, as well as these Checkly helpers: | Helper | Example | Description | | ---------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | `$UUID` | `{{$UUID}}` | Generates a random UUID v4, such as `9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d`. | | `$RANDOM_NUMBER` | `{{$RANDOM_NUMBER}}` | Generates a random integer from 0 through 10,000. | | `tagValue` | `{{tagValue TAGS "team" "UNKNOWN"}}` | Returns the value from the first matching `team:value` or `team=value` tag. Returns the optional final argument when no tag matches. | | `startsWith` | `{{startsWith value "prefix:"}}` | Returns whether a string starts with the supplied prefix. | | `replace` | `{{replace value "old" "new"}}` | Replaces the first matching string. | | `split` | `{{split "a,b,c" "," 1}}` | Splits a string and returns the item at the supplied zero-based index. | | `year` | `{{year}}` or `{{year 2}}` | Returns the current year with four or two digits. | | `date`, `moment` | `{{date "YYYY-MM-DD"}}` | Formats the current date. The two helper names are aliases. | #### Get a value from a tag Use `tagValue` when your tags store key-value metadata. Given these tags: ```text theme={null} team:payments environment=production ``` This template: ```handlebars theme={null} {{tagValue TAGS "team" "UNKNOWN"}} ``` renders `payments`. Both `:` and `=` separators are supported. If neither `team:...` nor `team=...` exists, the template renders the fallback `UNKNOWN`. If multiple tags match, Checkly returns the first one. Send a test message from the webhook editor to inspect the rendered request before saving your alert channel. #### Format dates and times The `date` and `moment` helpers support [Day.js-compatible format tokens](https://day.js.org/docs/en/display/format). You can format the current time, an absolute date, or relative phrases such as `yesterday`, `2 days ago`, and `last week`. * `{{date "YYYY-MM-DD"}}` formats the current date, for example as `2026-07-24`. * `{{date "2026-07-24" "MMMM DD, YYYY"}}` renders `July 24, 2026`. * `{{moment "2 days ago" "YYYY-MM-DD"}}` formats the date from two days ago. * `{{moment "last week" "X"}}` formats last week as a Unix timestamp. #### Compare values Use comparison helpers as blocks or subexpressions where applicable. Available helpers include: * Equality and ordering: `eq`, `is`, `isnt`, `gt`, `gte`, `lt`, `lte`, and `compare`. * Boolean logic: `and`, `or`, `not`, `neither`, `isTruthy`, and `isFalsey`. * Value checks: `contains`, `has`, and `default`. * Numeric conditions: `ifEven`, `ifOdd`, and `ifNth`. * Inverse comparisons: `unlessEq`, `unlessGt`, `unlessGteq`, `unlessLt`, and `unlessLteq`. For example: ```json theme={null} { "message": "Check is {{#eq ALERT_TYPE 'ALERT_FAILURE'}}DOWN{{/eq}} {{#eq ALERT_TYPE 'ALERT_RECOVERY'}}UP{{/eq}}" } ``` The above webhook body uses the `{{#eq}}` helper to execute the logic *If the ALERT\_TYPE variable equals 'ALERT\_FAILURE', print 'DOWN', if it equals 'ALERT\_RECOVERY' print 'UP'* So in the case of a failure event, the body would render to: ```json theme={null} { "message": "Check is DOWN" } ``` Two clear benefits here: * You only need to create one webhook to inform a 3rd party system. * You can translate Checkly terms to your 3rd party tool's term for the same concept, i.e. the "up status" of a check. ## Webhook Secrets You can validate each webhook we deliver to your endpoint(s). Using the optional webhook secret, you can: 1. Check if the webhook was sent by Checkly. 2. Check if the payload was not altered in any way during transmission. When you create a webhook secret, we proceed to use that secret token to cryptographically sign the webhook payload using the SHA256 hash algorithm. We add the resulting hash to the HTTP header `x-checkly-signature` on each webhook. On the receiving end, you can then use the value of the `x-checkly-signature` header to assert the validity and authenticity of the webhook and its payload. Have a look at the code examples below on how to use the header and your favourite web framework. ```javascript Node.js theme={null} // We store the webhook secret in an environment variable called CHECKLY_WEBHOOK_SECRET const app = require('express')(); const bodyParser = require('body-parser'); const crypto = require('crypto'); function isVerifiedPayload (payload, signature) { const secret = process.env.CHECKLY_WEBHOOK_SECRET const hmac = crypto.createHmac('sha256', secret) const digest = hmac.update(payload).digest('hex') return crypto.timingSafeEqual(Buffer.from(digest), Buffer.from(signature)) } app.post('/webhook', bodyParser.json({ type: 'application/json' }), (request, response) => { const signature = request.headers['x-checkly-signature']; const payload = JSON.stringify(request.body) if (isVerifiedPayload(payload, signature)) { console.log('Signature is valid') response.status(200).send(); } else { console.error('Signature does not match') response.status(400).send(); } }); app.listen(4242, () => console.log('Running on port 4242')) ``` ```ruby Ruby theme={null} # We store the webhook secret in an environment variable called CHECKLY_WEBHOOK_SECRET require 'sinatra' set :port, 4242 post '/webhook' do signature = request.env['HTTP_X_CHECKLY_SIGNATURE'] payload = request.body.read digest = OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha256'), ENV['CHECKLY_WEBHOOK_SECRET'], payload) if Rack::Utils.secure_compare(digest, signature) status 200 return else status 400 return end end ``` ```python Python theme={null} # This example assumes you use Django import hmac from hashlib import sha256 from django.conf import settings from django.http import HttpResponse, HttpResponseBadRequest from django.views.decorators.csrf import csrf_exempt from django.utils.encoding import force_bytes import json @csrf_exempt def webhook(request): signature = request.META.get('HTTP_X_CHECKLY_SIGNATURE') mac = hmac.new(settings.CHECKLY_WEBHOOK_SECRET.encode('utf-8'), msg=request.body, digestmod=sha256) print(mac.hexdigest()) if not hmac.compare_digest(mac.hexdigest(), signature): return HttpResponseBadRequest() return HttpResponse() ``` ## Webhook Retries Checkly will retry your webhook up to **5 times** if we get an HTTP response higher than 399, e.g. a 404 or 503. Each retry is backed off 20 seconds for a total retry period of `5 * 20 = 100 seconds`. This means that for checks on a 1 minute schedule, there is a potential overlap between a failure alert and recovery alert. For this reason every webhook we send has a timestamp in the `x-checkly-timestamp` header. You can use this timestamp on the receiving end to ignore any webhooks that come in "late". ## Webhook Examples The following examples give an idea how to integrate Checkly with 3rd party alerting and issue tracking systems. ### OpsGenie You can create an OpsGenie alert by POST-ing the following body ```json theme={null} { "message": "{{ALERT_TITLE}}", "description": "{{ALERT_TYPE}}
{{STARTED_AT}} ({{RESPONSE_TIME}}ms)
{{RESULT_LINK}}", "tags": [{{#each TAGS}} "{{this}}" {{#unless @last}},{{/unless}} {{/each}}] } ``` to the OpsGenie `alerts` API endpoint ``` https://{{OPSGENIE_API_KEY}}@api.opsgenie.com/v2/alerts ``` Or you can add the OpsGenie API key in the headers, e.g. ``` Authorization: GenieKey {{OPSGENIE_API_KEY}} ``` This is an example of a full alert body: ```json theme={null} { "message": "{{ALERT_TITLE}}", "description": "{{ALERT_TYPE}}: {{CHECK_NAME}}
{{STARTED_AT}} ({{RESPONSE_TIME}}ms)
{{RESULT_LINK}}", "responders": [ { "id":"4513b7ea-3b91-438f-b7e4-e3e54af9147c", "type":"team" } ], "tags": ["Critical", "Production"], "priority":"P1", "note": "Location: {{RUN_LOCATION}}" } ``` In case you would like different teams to be responsible for different Check Groups, you could add a `CHECK_GROUP_TEAM` variable with a different value for each Group, then modify the above snippet with the following: ```json theme={null} "responders": [ { "id":"{{CHECK_GROUP_TEAM}}", "type":"team" } ] ``` ### PagerDuty Given an existing service on your PagerDuty account, create an incident for it by posting the following body ```json theme={null} { "incident": { "type": "incident", "title": "{{ALERT_TITLE}}", "service": { "id": "", "type": "service_reference" }, "body": { "type": "incident_body", "details": "Check {{CHECK_NAME}} with ID {{CHECK_ID}} has failed from location {{RUN_LOCATION}}. See check result for details: {{RESULT_LINK}}" } } } ``` to `https://api.pagerduty.com/incidents`. You will need to set the following headers: pagerduty incident headers

### Pushover Send a message using [Pushover](https://pushover.net/) by posting this body: ```json theme={null} { "token":"YOUR_SECRET_TOKEN_FROM_PUSHOVER", "user":"YOUR_USER_FROM_PUSHOVER", "title":"{{ALERT_TITLE}}", "html":1, "priority":2, "retry":30, "expire":10800, "message":"{{ALERT_TYPE}}
{{STARTED_AT}} ({{RESPONSE_TIME}}ms)
{{RESULT_LINK}}" } ``` ### Trello You can create a [Trello](https://trello.com) card using just the URL and no payload: ``` https://api.trello.com/1/cards?idList=5b28c04aed47522097be8bc4&key={{TRELLO_KEY}}&token={{TRELLO_TOKEN}}&name={{CHECK_NAME}} ``` ### SSL alert You can send your SSL alerts using webhooks. Using the following body: ```json theme={null} { "message": "{{ALERT_TITLE}}", "link":"{{RESULT_LINK}}" } ``` Will yield the following output, where we customize the `ALERT_TITLE` to include the domain and the days remaining till your certificate expires. ```json theme={null} { "message": "The SSL certificate for api.checklyhq.com will expire in 14 days", "link": "http://app-test.checklyhq.com/checks/08437f9c-df8c-45ed-975a-a3f9e24d626d" } ``` ### Twilio You can configure a webhook to POST to a JavaScript snippet running in a [Twilio Function](https://www.twilio.com/docs/runtime/functions). This code receives the Checkly webhook JSON, then triggers a Twilio "Flow execution": ```javascript theme={null} //"From" is the sender phone number //"To" is the receiver phone number exports.handler = async function (context, event, callback) { const { From, To, Event, Link } = event; const client = context.getTwilioClient(); try { const execution = await client.studio.flows(FLOW_SID) .executions .create({ to: To, from: From, parameters: { Event, Link } }) console.log(`Created execution ${execution.sid}`); return callback(null, "OK"); } catch (error) { return callback(error); } }; ``` ### Jira A webhook can be used to create a new issue on Jira via the [Jira API](https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/), for example in the case of a previously passing check that switches to failing state. We will be creating a POST request to `{{JIRA_INSTANCE_URL}}/rest/api/3/issue`, where the content of your `JIRA_INSTANCE_URL` environment variable would look something like `https://your-jira-instance-name.atlassian.net`. The required headers will be: ```bash theme={null} Authorization: Basic Accept: application/json Content-Type: application/json ``` For more details on authenticating with the Jira API, refer to [Atlassian's guide on basic authentication](https://developer.atlassian.com/cloud/jira/software/basic-auth-for-rest-apis/). An example body could look as follows: ```json theme={null} { "fields": { "description": { // your Jira issue description, using Atlassian Document Format (ADF) "version": 1, "type": "doc", "content": [ { "type": "paragraph", "content": [ { "type": "text", "text": "View check result", "marks": [ { "type": "link", "attrs": { "href": "{{RESULT_LINK}}" } } ] } ] } ] }, "issuetype": { "id": "10001" // your Jira issue type id }, "labels": [ "needs_investigation" ], "priority": { // dynamically set the issue priority, based on the check's tags "id": {{#contains TAGS "P1"}} "1" {{else}} {{#contains TAGS "P2"}} "2" {{else}} {{#contains TAGS "P3"}} "3" {{else}} {{#contains TAGS "P4"}} "4" {{else}} {{#contains TAGS "P5"}} "5" {{else}} "3" {{/contains}} {{/contains}} {{/contains}} {{/contains}} {{/contains}} }, "project": { "key": "ABC" // your Jira project key }, "summary": "{{ALERT_TITLE}}" } } ``` For full details on creating issues via the Jira API, see [Atlassian's documentation for this endpoint](https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issues/#api-rest-api-3-issue-post). You can also use version 2 of the Jira API (i.e. `{{JIRA_INSTANCE_URL}}/rest/api/2/issue`). The only difference is that version 2 does not support [Atlassian Document Format (ADF)](https://developer.atlassian.com/cloud/jira/platform/apis/document/structure/). # Run Checkly from GitHub Actions Source: https://www.checklyhq.com/docs/integrations/ci-cd/github/actions Run Checkly test sessions from GitHub Actions and report results through workflow jobs or external GitHub Checks. Use the Checkly GitHub Action to run Monitoring as Code constructs from a repository or trigger checks already deployed to Checkly. The Action uses the Checkly CLI and can report the resulting test session through the GitHub Actions job or a separate GitHub Check. * A GitHub repository with GitHub Actions enabled. * A Checkly project that uses Checkly CLI `8.15.0` or newer. * `CHECKLY_API_KEY` stored as a GitHub Actions secret. * `CHECKLY_ACCOUNT_ID` stored as a GitHub Actions variable. Account IDs do not need to be secrets. * For detached GitHub Check reporting, the Checkly GitHub App connected to the same Checkly account and granted access to the repository. Install the GitHub App from the Checkly account that will run the test session. Go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations), click **Integrate with GitHub**, and grant the installation access to the repository. Installing the app directly from its GitHub listing does not connect the installation to your Checkly account. ## Run checks on pull requests Use this workflow when your Checkly constructs are defined in the repository: ```yaml .github/workflows/checkly.yml theme={null} name: Checkly on: pull_request: jobs: checkly: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: checkly/checkly-action@v1 with: command: test cli-version: latest install-command: npm ci reporting: auto env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` The checkout step makes the pull request's constructs and project dependencies available to the Action. With `reporting: auto`, the Action uses a detached GitHub Check when it is available. Otherwise, the workflow waits for the test session and reports the result through the GitHub Actions job. ## Choose what to run The `command` input selects the source of the checks: | Command | What it runs | | --------- | ----------------------------------------------------------------------------------------------------------------------- | | `test` | Constructs from the checked-out repository. Use this to validate pull request changes before deploying them to Checkly. | | `trigger` | Checks that are already deployed to the Checkly account. It does not load or test pull request-local constructs. | For `test`, use `tags`, `grep`, or `files` to select repository constructs: ```yaml theme={null} with: command: test tags: | production,webapp production,api grep: checkout files: | __checks__/**/*.check.ts ``` Each `tags` line creates a separate `--tags` filter. Comma-separated tags on one line must all match. For `trigger`, select deployed checks with tags or check IDs: ```yaml .github/workflows/checkly-trigger.yml theme={null} name: Checkly trigger on: workflow_dispatch: jobs: checkly: runs-on: ubuntu-latest steps: - uses: checkly/checkly-action@v1 with: command: trigger cli-version: latest reporting: auto tags: production env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` ## Choose a reporting mode The `reporting` input controls whether the workflow waits for the Checkly test session: | Value | Behavior | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `auto` | Use a detached Checkly run and external GitHub Check when available. Otherwise, wait and report through the GitHub Actions job. This is the default. | | `github-check` | Require detached GitHub Check reporting. The Action fails preflight before running checks when the Checkly GitHub App cannot report to the repository. | | `github-actions` | Always wait for the test session. The workflow job carries the result and the Action writes the Checkly report to the job summary. | Detached reporting produces two separate GitHub Check Runs: the GitHub Actions job and the external Checkly GitHub Check. The workflow job can finish successfully after starting the detached test session while the external Checkly GitHub Check is still running. Use `github-check-name` to set the external GitHub Check Run name. It defaults to `Checkly`; it is not the name of a Checkly check. ```yaml theme={null} with: command: test reporting: github-check github-check-name: Checkly PR validation ``` A GitHub Actions job Check Run, an external Checkly GitHub Check Run, and a Checkly check are different objects. The Action does not publish a legacy commit status. The Checkly test session contains the execution results behind either reporting mode. ## Resolve the CLI and dependencies The Action requires Checkly CLI `8.15.0` or newer. Use `cli-version: latest` unless you need to test against a specific stable release. The Action resolves the CLI after running `install-command` inside `working-directory`: 1. If the project installs the `checkly` package, the Action uses that local CLI so the CLI and imported constructs share the same module installation. 2. Otherwise, the Action runs `npx checkly@`. Keep the project's `checkly` dependency at `8.15.0` or newer. If you use an exact stable `cli-version`, it must match the locally installed Checkly version. In a monorepo, `working-directory` selects where the Action resolves the config, dependencies, and CLI. When dependencies install from the repository root, install them before invoking the Action: ```yaml .github/workflows/checkly-monorepo.yml theme={null} name: Checkly webapp on: pull_request: jobs: checkly: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - uses: checkly/checkly-action@v1 with: command: test cli-version: latest working-directory: apps/webapp reporting: auto env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` If the target directory has its own package manifest and lockfile, use `install-command` instead. The command runs inside `working-directory` before the Action resolves the local CLI. ## Report to the correct commit GitHub Check Runs attach to commits. By default, the Action uses: | Event | Default commit | | --------------------------------------- | -------------------------- | | `pull_request` or `pull_request_target` | The pull request head SHA. | | Any other event | `GITHUB_SHA`. | Set `github-sha` when the commit you check out and test differs from that default. This commonly applies to `repository_dispatch` and deployment workflows. Use the same SHA for checkout and Check reporting: ```yaml .github/workflows/checkly-preview-dispatch.yml theme={null} name: Validate preview on: repository_dispatch: types: [preview-ready] jobs: checkly: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: ref: ${{ github.event.client_payload.sha }} - uses: checkly/checkly-action@v1 with: command: test cli-version: latest install-command: npm ci reporting: auto github-sha: ${{ github.event.client_payload.sha }} github-check-name: Checkly preview validation env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ENVIRONMENT_URL: ${{ github.event.client_payload.environment_url }} ``` The commit must belong to the current repository. `github-sha` only selects the commit that receives the external Check; it does not grant repository access. Checkly still verifies that the GitHub App installation for the Checkly account can access the repository. ## Test preview deployments Pull request and custom-dispatch workflows usually need to pass their preview URL explicitly as `ENVIRONMENT_URL`, as shown in the `repository_dispatch` example above. For `deployment_status` events, the Action automatically exposes `github.event.deployment_status.environment_url` as `ENVIRONMENT_URL` when the workflow has not already set it. GitHub sets `GITHUB_SHA` to the commit being deployed, so the Action can normally use its default commit metadata: ```yaml .github/workflows/checkly-deployment.yml theme={null} name: Validate deployment on: deployment_status: jobs: checkly: if: github.event.deployment_status.state == 'success' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: checkly/checkly-action@v1 with: command: test cli-version: latest install-command: npm ci reporting: auto env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` If a custom deployment workflow checks out a commit from an event payload instead of `GITHUB_SHA`, pass that same commit through `github-sha`. If you want Checkly itself to listen for GitHub deployment events and trigger checks linked to a repository, use [GitHub deployment hooks](/docs/integrations/ci-cd/github/deployments) instead of this workflow. ## Protect merges with detached checks When you use detached reporting, require the external Checkly GitHub Check in your branch rules. Requiring only the GitHub Actions job does not wait for the external Checkly test session to finish. Configure the exact `github-check-name` as a [required check](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/troubleshooting-required-status-checks). If the workflow uses path filters, make sure it still creates the required Checkly Check for every pull request commit. Prefer Action inputs such as `files`, `grep`, or `tags` when you need to select a smaller test set. ## Next steps * Learn more about [`checkly test`](/docs/cli/checkly-test). * Learn more about [`checkly trigger`](/docs/cli/checkly-trigger). * [Deploy scoped Checkly secrets from an external secrets manager](/docs/platform/manage-secrets-at-scale). * Use [GitHub deployment hooks](/docs/integrations/ci-cd/github/deployments) when you want Checkly to react directly to GitHub deployment events. # GitHub deployment hooks Source: https://www.checklyhq.com/docs/integrations/ci-cd/github/deployments Trigger linked Checkly checks from GitHub deployment events and use deployment environment URLs as test targets. Use GitHub deployment hooks when you want Checkly to listen for GitHub `deployment_status` events and trigger checks linked to a repository. This is separate from the [Checkly GitHub Action](/docs/integrations/ci-cd/github/actions), which starts test sessions from a GitHub Actions workflow. * The Checkly GitHub integration installed from the [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) page of the Checkly account that owns the checks. * A deployment provider that sends successful GitHub `deployment_status` events, such as Vercel for GitHub or Heroku Pipelines. * One or more Checkly checks linked to the GitHub repository from the check's **CI/CD** settings. Start the installation from [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) in Checkly and click **Integrate with GitHub**. Grant the installation access to the repository you want to link. Installing the app directly from GitHub does not connect it to the Checkly account, so deployment hooks and GitHub Check reporting will not work correctly. GitHub `deployment_status` webhook events are triggered whenever a deployment service reports a successful deployment. This works with services that report successful deployment statuses through the GitHub Deployments API, including: * [Vercel for GitHub](https://vercel.com/docs/v2/git-integrations/vercel-for-github) * [Heroku Pipelines](https://devcenter.heroku.com/articles/pipelines) * Other deployment services that use the [GitHub deployments API](https://docs.github.com/en/rest/deployments/deployments) and report successful deployment statuses. This integration has some [known limitations](/docs/integrations/ci-cd/github/deployments#limitations). Use the [Checkly GitHub Action](/docs/integrations/ci-cd/github/actions) when you want GitHub Actions to start a recorded test session and report the result on a pull request. ## Set up GitHub deployment hooks 1. Go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) and click **Integrate with GitHub**. 2. In GitHub, choose the account where you want to install the Checkly GitHub App and grant it access to the repositories you want to connect. After installation, GitHub redirects you to Checkly. 3. Open the check or check group you want to trigger and go to its **CI/CD** settings. 4. Click **Link GitHub repo** and select the repository that reports your deployment events. After you link the repository, Checkly listens for successful deployment events and runs the linked check or check group. You can also choose a run location, filter deployments by environment or environment URL, and include screenshots in the pull request comment. You can link multiple checks to the same repository. Checkly runs the linked checks as a test suite when GitHub reports a successful deployment. ## Manage deployment hooks with the API Use the Checkly Public API to create GitHub deployment hooks programmatically. A deployment trigger represents the link between one GitHub repository and either a check or a check group. * [Create a deployment trigger](/docs/api-reference/deployment-triggers/create-a-deployment-trigger) for a check or check group. Repeating the request for the same target returns the existing trigger without changing its settings. * [List deployment triggers](/docs/api-reference/deployment-triggers/list-all-deployment-triggers) for your account. The API supports GitHub deployment triggers. To configure the Checkly Vercel integration, use the [Vercel integration settings](/docs/integrations/ci-cd/vercel/overview). If you omit both deployment filters, every successful deployment reported for the repository can trigger a run, including production deployments. Set `environmentRegexFilter` or `environmentUrlRegexFilter` when you only want to run checks against specific environments. You can use deployment environment URLs with API, URL, browser, and multi-step checks. API and URL checks replace the host in the configured request URL. Browser and multi-step checks receive the deployment URL through the `ENVIRONMENT_URL` environment variable. To change or remove a trigger created through the API, open the check or check group in Checkly and use its **CI/CD** settings. ## Use deployment environment URLs GitHub reports an **environment URL** on each deployment event. Depending on what deployment service you use, this environment URL can be used to run your check on different target environments than configured in your check. The primary use case for this is validating temporary review/branch deployments, such as those provided by Vercel and Heroku Pipelines, before going to production. So, when you enable the **"Use environment URL from deployment trigger"** checkbox there are two scenarios: ### API checks & environment URLs With API checks, Checkly replaces the hostname part of your request URL with the host in the environment URL. For example: * Your configured URL: `https://api.acme.com/v1/customers?page=1` * Environment URL: `https://now.customer-api.qis6va2z7.now.sh` * Replaced URL: `https://now.customer-api.qis6va2z7.now.sh/v1/customers?page=1` Notice we only replace the **host** part, not the URL path or any query parameters. ### Browser checks & environment URLs For browser checks, the environment URL is exposed as the `ENVIRONMENT_URL` environment variable. This means you can use that variable in your code to replace any hardcoded URL you might have, i.e.: ```typescript test.spec.ts theme={null} import { expect, test } from '@playwright/test' test('assert response status of page', async ({ page }) => { const targetUrl = process.env.ENVIRONMENT_URL || 'https://www.checklyhq.com' const response = await page.goto(targetUrl) expect(response.status()).toBeLessThan(400) }) ``` ```javascript test.spec.js theme={null} const { expect, test } = require('@playwright/test') test('assert response status of page', async ({ page }) => { const targetUrl = process.env.ENVIRONMENT_URL || 'https://www.checklyhq.com' const response = await page.goto(targetUrl) expect(response.status()).toBeLessThan(400) }) ``` This sets the `targetUrl` variable to either the `ENVIRONMENT_URL` or your main production URL. When a **Preview** deployment happens on GitHub, this check runs the script against the preview environment. This check also runs on a set interval and checks your production environment. With one check, you can validate preview deployments from GitHub events and keep running the same check on your production schedule. ## Limitations GitHub deployment hooks lack support for several features: * Client certificates are not applied. * OpenTelemetry integration headers are not applied. * Private locations are not available. Use the [Checkly GitHub Action](/docs/integrations/ci-cd/github/actions) or the [Checkly CLI](/docs/cli/overview) in your CI/CD pipeline when you need these features. # Deploy Checkly secrets from AWS Secrets Manager Source: https://www.checklyhq.com/docs/integrations/ci-cd/github/secrets/aws-secrets-manager Use GitHub Actions and OIDC to deploy scoped Checkly secrets from AWS Secrets Manager. Use GitHub OpenID Connect (OIDC) to obtain short-lived AWS credentials, read the secrets required by one Checkly CLI project, and deploy them as group- or check-level secrets. For the architecture and scoping model, see [Manage secrets at scale](/docs/platform/manage-secrets-at-scale). * A Checkly CLI project with `checkly` installed as a dependency. * An AWS account with your values stored in AWS Secrets Manager. * A GitHub repository with Actions enabled. * A protected GitHub environment named `checkly-production`. Restrict its deployment branches and reviewers as appropriate. * The repository or environment variables `AWS_ROLE_ARN`, `AWS_REGION`, and `CHECKLY_ACCOUNT_ID`. * Permission to create an AWS IAM OIDC provider and role. This guide uses a fictional payments service; replace its names, paths, and URL with your own application values. Checkly requires `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID`; `AWS_ROLE_ARN` and `AWS_REGION` configure this workflow. ## Configure AWS access Follow AWS's guide for [using Secrets Manager in GitHub jobs](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_github.html) to create an OIDC provider and IAM role. Restrict the role's trust policy to the repository and protected GitHub environment that deploy Checkly. Grant the role `secretsmanager:GetSecretValue` only for the secrets listed in this workflow. The retrieval action also requires `secretsmanager:ListSecrets`; secrets encrypted with a customer-managed KMS key require `kms:Decrypt` on that key. Store the role ARN and region in `AWS_ROLE_ARN` and `AWS_REGION` GitHub Actions variables. ## Scope secrets in your constructs Use `secret()` to read `CHECKLY_SECRET_*` from the deploy process. Set `secret: true` so Checkly stores the value as a non-retrievable secret. Attach each value at the narrowest level that needs it: ```typescript checks/payments.check.ts theme={null} import { ApiCheck, CheckGroupV2, Frequency } from "checkly/constructs"; import { secret } from "checkly/util"; const paymentsGroup = new CheckGroupV2("payments", { name: "Payments", activated: true, muted: false, frequency: Frequency.EVERY_10M, locations: ["us-east-1"], environmentVariables: [ { key: "PAYMENTS_API_KEY", value: secret("PAYMENTS_API_KEY"), secret: true, }, ], }); new ApiCheck("payments-admin-api", { name: "Payments admin API", group: paymentsGroup, environmentVariables: [ { key: "PAYMENTS_ADMIN_TOKEN", value: secret("PAYMENTS_ADMIN_TOKEN"), secret: true, }, ], request: { method: "GET", url: "https://payments.example.com/admin/health", headers: [ { key: "Authorization", value: "Bearer {{PAYMENTS_API_KEY}}" }, { key: "X-Admin-Token", value: "{{PAYMENTS_ADMIN_TOKEN}}" }, ], }, }); ``` `secret('PAYMENTS_API_KEY')` reads `CHECKLY_SECRET_PAYMENTS_API_KEY` during deployment and fails if it is missing. The shorter key `PAYMENTS_API_KEY` is what your check receives at runtime. ## Fetch and deploy in GitHub Actions Fetch each AWS secret with an explicit `CHECKLY_SECRET_*` alias. The retrieval action masks the values and adds them to the environment of subsequent steps. ```yaml .github/workflows/checkly-deploy.yml theme={null} name: Deploy Checkly on: push: branches: [main] workflow_dispatch: jobs: deploy: runs-on: ubuntu-latest environment: checkly-production permissions: contents: read id-token: write steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - name: Install dependencies run: npm ci - name: Authenticate to AWS uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: ${{ vars.AWS_ROLE_ARN }} aws-region: ${{ vars.AWS_REGION }} allowed-account-ids: "123456789012" mask-aws-account-id: true role-session-name: checkly-${{ github.run_id }} - name: Fetch deployment secrets uses: aws-actions/aws-secretsmanager-get-secrets@v3 with: secret-ids: | CHECKLY_API_KEY, checkly/production/api-key CHECKLY_SECRET_PAYMENTS_API_KEY, checkly/production/payments-api-key CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN, checkly/production/payments-admin-token - name: Deploy Checkly env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} run: npx checkly deploy --force ``` Install dependencies before fetching secrets, and keep deployment as the final step. Values created by `aws-secretsmanager-get-secrets` are available to every later step in the job, not only the deploy command. Do not print or persist fetched values. Keep deployment as the final step that receives them. ## Rotate a secret When you rotate a value in AWS Secrets Manager, run this workflow again. Checkly receives a copy during `checkly deploy`; rotation in AWS does not update an already deployed Checkly secret automatically. # Deploy Checkly secrets from Azure Key Vault Source: https://www.checklyhq.com/docs/integrations/ci-cd/github/secrets/azure-key-vault Authenticate to Azure with GitHub OIDC, load deployment secrets from Azure Key Vault, and deploy them at group or check scope. Use GitHub OpenID Connect (OIDC) to give a trusted deployment job short-lived access to Azure Key Vault. The job maps each Key Vault secret to the `CHECKLY_SECRET_*` name expected by your Checkly constructs, then runs `checkly deploy`. For the scoping model and operational guidance behind this workflow, read [Manage secrets at scale](/docs/platform/manage-secrets-at-scale). * A Checkly Monitoring as Code project that declares secrets with `secret()` and `secret: true` at [group or check scope](/docs/platform/manage-secrets-at-scale#choose-the-narrowest-scope). * An Azure Key Vault that uses the Azure role-based access control (RBAC) permission model. * A Microsoft Entra application or user-assigned managed identity with a [federated identity credential for GitHub Actions](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust). * The identity has the `Key Vault Secrets User` role on the vault that contains this project's secrets. * `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `AZURE_KEY_VAULT`, and `CHECKLY_ACCOUNT_ID` configured as GitHub Actions variables. * The Checkly API key stored in Key Vault as `checkly-api-key`, alongside the secrets required by this project. This guide uses a fictional payments service; replace its names and URL with your own application values. Checkly requires `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID`; the `AZURE_*` variables configure this workflow. ## Configure Azure access Follow Microsoft's [GitHub OIDC setup guide](https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure-openid-connect) to create a federated identity for the repository and GitHub environment that deploy Checkly. Assign the identity the read-only [`Key Vault Secrets User` role](https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide#azure-built-in-roles-for-key-vault-data-plane-operations) at the narrowest practical scope. The GitHub Actions runner must be able to reach the vault. ## Map Key Vault secrets to Checkly The following example assumes your application uses two credentials stored in Key Vault as `payments-api-key` and `payments-admin-token`. These are example names, not Azure or Checkly requirements. Reference their deployment environment names in your constructs: ```typescript __checks__/payments.check.ts theme={null} import { ApiCheck, CheckGroupV2 } from 'checkly/constructs' import { secret } from 'checkly/util' const paymentsGroup = new CheckGroupV2('payments', { name: 'Payments', environmentVariables: [ { key: 'PAYMENTS_API_KEY', value: secret('PAYMENTS_API_KEY'), secret: true, }, ], }) new ApiCheck('payments-admin-health', { name: 'Payments admin health', group: paymentsGroup, environmentVariables: [ { key: 'PAYMENTS_ADMIN_TOKEN', value: secret('PAYMENTS_ADMIN_TOKEN'), secret: true, }, ], request: { method: 'GET', url: 'https://payments.example.com/admin/health', headers: [ { key: 'Authorization', value: 'Bearer {{PAYMENTS_ADMIN_TOKEN}}' }, ], }, }) ``` `secret('PAYMENTS_API_KEY')` reads `CHECKLY_SECRET_PAYMENTS_API_KEY` when the CLI loads the project. The Key Vault name and the Checkly environment name do not need to match. ## Deploy from GitHub Actions This workflow installs project dependencies before authenticating to Azure. It retrieves named secrets with [`az keyvault secret show`](https://learn.microsoft.com/en-us/cli/azure/keyvault/secret#az-keyvault-secret-show), masks each retrieved value, exports it only in the deployment shell, and does not write it to `GITHUB_ENV`. ```yaml .github/workflows/deploy-checkly.yml theme={null} name: Deploy Checkly on: push: branches: [main] workflow_dispatch: permissions: contents: read jobs: deploy: runs-on: ubuntu-latest environment: production permissions: contents: read id-token: write env: AZURE_CORE_OUTPUT: none steps: - uses: actions/checkout@v7 with: persist-credentials: false - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - name: Install dependencies run: npm ci - name: Sign in to Azure with OIDC uses: azure/login@v3 with: client-id: ${{ vars.AZURE_CLIENT_ID }} tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} - name: Load secrets and deploy shell: bash env: AZURE_KEY_VAULT: ${{ vars.AZURE_KEY_VAULT }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} run: | set -euo pipefail get_secret() { az keyvault secret show \ --vault-name "$AZURE_KEY_VAULT" \ --name "$1" \ --query value \ --output tsv } mask_secret() { local value="${1//%/%25}" value="${value//$'\r'/%0D}" value="${value//$'\n'/%0A}" printf '::add-mask::%s\n' "$value" } CHECKLY_API_KEY="$(get_secret checkly-api-key)" CHECKLY_SECRET_PAYMENTS_API_KEY="$(get_secret payments-api-key)" CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN="$(get_secret payments-admin-token)" mask_secret "$CHECKLY_API_KEY" mask_secret "$CHECKLY_SECRET_PAYMENTS_API_KEY" mask_secret "$CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN" export CHECKLY_API_KEY export CHECKLY_SECRET_PAYMENTS_API_KEY export CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN npx checkly deploy --force ``` The job grants only `contents: read` and `id-token: write`. The latter lets `azure/login` request a GitHub OIDC token; it does not grant repository write access. `AZURE_CORE_OUTPUT: none` prevents accidental Azure CLI output, while `--output tsv` explicitly returns only each requested value to the shell. Do not print or persist fetched values. This example assumes token-like, single-line secret values. ## Rotate a secret Rotating a value in Azure Key Vault does not automatically update the copy stored by Checkly. Run this trusted deployment workflow again after rotation. The next deployment fetches the current value returned by Key Vault and updates the scoped Checkly secret. # Deploy Checkly secrets from Google Secret Manager Source: https://www.checklyhq.com/docs/integrations/ci-cd/github/secrets/google-secret-manager Use GitHub OIDC to fetch Google Secret Manager values and deploy them as scoped Checkly secrets. Use Workload Identity Federation to authenticate a trusted GitHub Actions deployment to Google Cloud without storing a service account key in GitHub. The workflow fetches only the values its Checkly project needs and exposes them to the Checkly deploy step. For guidance on choosing account, group, or check scope, see [Manage secrets at scale](/docs/platform/manage-secrets-at-scale). * A Checkly Monitoring as Code project with its dependencies and lockfile committed. * `CHECKLY_ACCOUNT_ID` stored as a GitHub Actions variable. * The Checkly API key and required deployment values stored as enabled secret versions in Google Secret Manager. * A Google Cloud project with the IAM, Resource Manager, Service Account Credentials, Security Token Service, and Secret Manager APIs enabled. * Permission to configure Workload Identity Federation, service account impersonation, and IAM access on the required secrets. This guide uses a fictional payments service; replace its names and URL with your own application values. Checkly requires `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID`; the `GCP_*` variables configure this workflow. ## Configure Google Cloud access Follow Google's [deployment pipeline guide](https://cloud.google.com/iam/docs/workload-identity-federation-with-deployment-pipelines) or the [`google-github-actions/auth` instructions](https://github.com/google-github-actions/auth#workload-identity-federation-through-a-service-account) to configure Workload Identity Federation and a dedicated service account. Restrict the Workload Identity Provider to the repository and ref that deploy Checkly. Grant the repository principal `roles/iam.workloadIdentityUser` on the service account, then grant that service account `roles/secretmanager.secretAccessor` only on the secrets listed in the workflow. Add these non-secret values as GitHub Actions variables: | Variable | Value | | -------------------------------- | --------------------------------------------------------------------------------------- | | `GCP_PROJECT_ID` | The project ID that contains the secrets. | | `GCP_WORKLOAD_IDENTITY_PROVIDER` | The full provider resource name, including the project **number** and `/providers/...`. | | `GCP_SERVICE_ACCOUNT` | The dedicated deployment service account email. | ## Scope secrets in your constructs Read each value with `secret()` and mark it as secret on the group or check that needs it: ```typescript __checks__/payments.check.ts theme={null} import { ApiCheck, CheckGroupV2 } from 'checkly/constructs' import { secret } from 'checkly/util' const paymentsGroup = new CheckGroupV2('payments', { name: 'Payments', environmentVariables: [ { key: 'PAYMENTS_API_KEY', value: secret('PAYMENTS_API_KEY'), secret: true, }, ], }) new ApiCheck('payments-admin', { name: 'Payments admin', group: paymentsGroup, environmentVariables: [ { key: 'PAYMENTS_ADMIN_TOKEN', value: secret('PAYMENTS_ADMIN_TOKEN'), secret: true, }, ], request: { method: 'GET', url: 'https://payments.example.com/admin', headers: [ { key: 'Authorization', value: 'Bearer {{PAYMENTS_ADMIN_TOKEN}}' }, ], }, }) ``` The helper reads `CHECKLY_SECRET_PAYMENTS_API_KEY` and `CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN` when the project is loaded for deployment. If a required value is missing, `secret()` stops the deployment instead of deploying an empty value. Checkly then stores each value at the declared scope. ## Fetch and deploy from GitHub Actions The Google Secret Manager action returns masked step outputs. Map those outputs to `CHECKLY_SECRET_*` variables only on the deploy step: Google maintains `get-secretmanager-secrets` in the `google-github-actions` organization, but its README states that it is not an officially supported Google Cloud product and is not covered by a Google Cloud support contract. ```yaml .github/workflows/deploy-checkly.yml theme={null} name: Deploy Checkly on: push: branches: - main workflow_dispatch: permissions: contents: read id-token: write jobs: deploy: if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - run: npm ci - id: google-auth name: Authenticate to Google Cloud uses: google-github-actions/auth@v3 with: project_id: ${{ vars.GCP_PROJECT_ID }} workload_identity_provider: ${{ vars.GCP_WORKLOAD_IDENTITY_PROVIDER }} service_account: ${{ vars.GCP_SERVICE_ACCOUNT }} - id: google-secrets name: Fetch deployment secrets uses: google-github-actions/get-secretmanager-secrets@v3 with: secrets: |- checkly_api_key:${{ vars.GCP_PROJECT_ID }}/checkly-api-key payments_api_key:${{ vars.GCP_PROJECT_ID }}/payments-api-key payments_admin_token:${{ vars.GCP_PROJECT_ID }}/payments-admin-token - name: Deploy Checkly run: npx checkly deploy --force env: CHECKLY_API_KEY: ${{ steps.google-secrets.outputs.checkly_api_key }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_SECRET_PAYMENTS_API_KEY: ${{ steps.google-secrets.outputs.payments_api_key }} CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN: ${{ steps.google-secrets.outputs.payments_admin_token }} ``` The `id-token: write` permission lets the job request a GitHub OIDC token. Google Cloud IAM still determines which secrets it can read. Do not print or persist fetched values. Expose the outputs only to the Checkly deployment step. ## Rotate a secret The short secret names in the workflow fetch the `latest` enabled version. After you add a new version in Google Secret Manager, run this deployment workflow again to update the encrypted value stored by Checkly. Rotation in Google Cloud alone does not update an already deployed Checkly secret. When the workflow no longer needs a value, remove it from the construct, workflow mapping, and service account IAM policy. # Deploy Checkly secrets from HashiCorp Vault Source: https://www.checklyhq.com/docs/integrations/ci-cd/github/secrets/hashicorp-vault Use GitHub OIDC to fetch HashiCorp Vault values and deploy them as scoped Checkly secrets. Use GitHub OpenID Connect (OIDC) to authenticate a trusted deployment job to HashiCorp Vault without storing a Vault token in GitHub. The job reads only the values required by one Checkly CLI project and exposes them to the Checkly deploy step. For guidance on choosing account, group, or check scope, see [Manage secrets at scale](/docs/platform/manage-secrets-at-scale). * A Checkly Monitoring as Code project with its dependencies and lockfile committed. * A GitHub repository with Actions enabled and a protected environment named `checkly-production`. * A Vault server reachable from the GitHub Actions runner, with the required values stored in a KV secrets engine. * Permission to configure a Vault JWT auth method, policy, and role. * `VAULT_ADDR` and `CHECKLY_ACCOUNT_ID` configured as GitHub Actions variables. * The Checkly API key stored in Vault alongside the secrets required by this CLI project. This guide uses a fictional payments service; replace its names, paths, and URL with your own application values. Checkly requires `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID`; `VAULT_ADDR` configures this workflow. ## Configure Vault access Configure Vault's JWT authentication for GitHub Actions and create a role for this deployment by following HashiCorp's [`vault-action` JWT guide](https://github.com/hashicorp/vault-action#jwt-with-github-oidc-tokens). The workflow below assumes that role is named `checkly-payments-deploy`. Restrict the role to the repository, branch, and GitHub environment that deploy Checkly. Its policy should grant read-only access only to the Vault paths referenced by this project. The role's audience must match `jwtGithubAudience` in the workflow. See HashiCorp's [JWT role reference](https://developer.hashicorp.com/vault/api-docs/auth/jwt#createupdate-role) for the Vault configuration options. ## Scope secrets in your constructs Read each value with `secret()` and mark it as secret on the group or check that needs it: ```typescript __checks__/payments.check.ts theme={null} import { ApiCheck, CheckGroupV2 } from "checkly/constructs"; import { secret } from "checkly/util"; const paymentsGroup = new CheckGroupV2("payments", { name: "Payments", environmentVariables: [ { key: "PAYMENTS_API_KEY", value: secret("PAYMENTS_API_KEY"), secret: true, }, ], }); new ApiCheck("payments-admin", { name: "Payments admin", group: paymentsGroup, environmentVariables: [ { key: "PAYMENTS_ADMIN_TOKEN", value: secret("PAYMENTS_ADMIN_TOKEN"), secret: true, }, ], request: { method: "GET", url: "https://payments.example.com/admin", headers: [ { key: "Authorization", value: "Bearer {{PAYMENTS_ADMIN_TOKEN}}" }, ], }, }); ``` The helper reads `CHECKLY_SECRET_PAYMENTS_API_KEY` and `CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN` when the CLI loads the project. If a required value is missing, `secret()` stops the deployment instead of deploying an empty value. ## Fetch and deploy from GitHub Actions The official HashiCorp action authenticates with GitHub's OIDC token and returns masked step outputs. Set `exportEnv: false`, then map the outputs to `CHECKLY_SECRET_*` variables only on the deploy step: ```yaml .github/workflows/deploy-checkly.yml theme={null} name: Deploy Checkly on: push: branches: [main] workflow_dispatch: jobs: deploy: if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest environment: checkly-production permissions: contents: read id-token: write steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - name: Install dependencies run: npm ci - id: vault-secrets name: Fetch deployment secrets uses: hashicorp/vault-action@v4 with: url: ${{ vars.VAULT_ADDR }} method: jwt role: checkly-payments-deploy jwtGithubAudience: https://github.com/YOUR_ORG exportEnv: false secrets: | secret/data/checkly/production/payments/checkly checkly_api_key | CHECKLY_API_KEY ; secret/data/checkly/production/payments/group payments_api_key | CHECKLY_SECRET_PAYMENTS_API_KEY ; secret/data/checkly/production/payments/admin-check admin_token | CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN - name: Deploy Checkly env: CHECKLY_API_KEY: ${{ steps.vault-secrets.outputs.CHECKLY_API_KEY }} CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_SECRET_PAYMENTS_API_KEY: ${{ steps.vault-secrets.outputs.CHECKLY_SECRET_PAYMENTS_API_KEY }} CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN: ${{ steps.vault-secrets.outputs.CHECKLY_SECRET_PAYMENTS_ADMIN_TOKEN }} run: npx checkly deploy --force ``` The job needs `id-token: write` to request the GitHub OIDC token and `contents: read` for checkout. The Vault role and policy determine which values the job can read. Do not print or persist fetched values. Install dependencies before fetching secrets and expose the outputs only to the Checkly deployment step. ## Rotate a secret Rotating a value in Vault does not automatically update the encrypted copy stored by Checkly. Run this trusted deployment workflow again after rotation. The action reads the current KV value and `checkly deploy` updates the secret at its declared group or check scope. When the workflow no longer needs a value, remove it from the construct, workflow mapping, and Vault policy or path. # Integrating Checkly in GitLab CI/CD Source: https://www.checklyhq.com/docs/integrations/ci-cd/gitlab/overview Run and deploy Checkly checks from a GitLab CI/CD pipeline, using a branch-aware setup that only promotes checks to monitors after they pass against production. ## A Basic GitLab Pipeline Example Create a new `.gitlab-ci.yml` file in your repo, or add the steps and stages from the example below to your existing file. This pipeline is "branch aware" and treats the `main` branch as the production branch. This means checks are only deployed to Checkly after they are ran against production (after merging to `main`) and the checks passed. ```yml .gitlab-ci.yml theme={null} image: node:latest # define your stages to deploy your app, run tests and then deploy the tests as monitors. stages: - deploy - checkly-test - checkly-deploy # Set the necessary credentials and export variables we can use to instrument our test run. Use the ENVIRONMENT_URL # to run your checks against staging, preview or production. variables: CHECKLY_API_KEY: "$CHECKLY_API_KEY" CHECKLY_ACCOUNT_ID: "$CHECKLY_ACCOUNT_ID" CHECKLY_TEST_REPO_BRANCH: "$CI_COMMIT_BRANCH" CHECKLY_TEST_REPO_URL: "$CI_PROJECT_URL" deploy-app: stage: deploy script: - echo "add your deployment logic here" # - ENVIRONMENT_URL= Pipelines tab will now look similar to this: GitLab CI Pipeline

# Integrating Checkly in Jenkins Source: https://www.checklyhq.com/docs/integrations/ci-cd/jenkins/overview Run the Checkly CLI from a Jenkins pipeline, setting API credentials, choosing a reporter, and deploying checks as monitors after they pass against production. We've optimized the [Checkly CLI](/docs/cli/overview/) to work in any CI/CD workflow. Here are the basics you need to know that will come in handy when adapting the examples we give you to your own, specific setup. 1. For **authentication**, make sure to set the `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` parameters as environment variables in your CI/CD platform. 2. Set the **reporter** you want to use for the `test` command using the `--reporter` flag, i.e. `--reporter=dot`. 3. Use `npx checkly test` to validate your checks before deploying. Test results, traces, and videos are available in Checkly as a **test session**. 4. Use the `--force` flag on the `deploy` and / or `destroy` commands to skip the normal confirmation steps. The CLI will attempt to parse `git` specific information from the environment to display in the test session as metadata. You can also set these data items specifically by using environment variables. | Item | Auto | Variable | Description | | ------------------ | ----- | ------------------------------------------------------ | ------------------------------------------- | | **Repository** | false | `repoUrl` in `checkly.config.ts` or `CHECKLY_REPO_URL` | The URL of your repo on GitHub, GitLab etc. | | **Commit hash** | true | `CHECKLY_REPO_SHA` | The SHA of the commit. | | **Branch** | true | `CHECKLY_REPO_BRANCH` | The branch name. | | **Commit owner** | true | `CHECKLY_REPO_COMMIT_OWNER` | The committer's name or email. | | **Commit message** | true | `CHECKLY_REPO_COMMIT_MESSAGE` | The commit message. | | **Environment** | false | `CHECKLY_TEST_ENVIRONMENT` | The environment name, e.g. "staging" | ## A Basic pipeline example Create a new Jenkinsfile in your repo, or add the steps and stages from the example below to your existing file. This pipeline is "branch aware" and treats the `main` branch as the production branch. This means checks are only deployed to Checkly after they are ran against production (after merging to `main`) and the checks passed. ## Configuring Jenkins to run the Checkly CLI As the Checkly CLI is a Node.js project, the main step you need to take is install the NodeJS plugin. 1. In Jenkins, go to **Manage Jenkins → Plugins → Available plugins** and look for the NodeJS plugin and install it. 2. After installing the NodeJS plugin, we need to configure it. Head over to **Manage Jenkins → Tools** and click "Add NodeJS" Jenkins Nodejs config

We recommend using any Node.js stable version higher than 16.x. ## Set your Checkly credentials Navigate to **Manage Jenkins → Manage Credentials** to add your Checkly account ID and API key to your preferred scope. Store them as "secret text" and assign and ID to the credential. Jenkins Nodejs config

## Configuring the Jenkins Pipeline Add the `Jenkinsfile` to your repo that defines the basic stages and steps. Make sure to set up your SCM settings correctly so Jenkins can fetch your git repo and look for the `Jenkinsfile` in the root of your project. Jenkins Nodejs config

The actual contents of your `Jenkinsfile` will differ based on your code and how you deploy. But in general, your pipeline should look as follows: 1. You deploy your application first. 2. You install the required dependencies for the Checkly CLI. 3. You run the `checkly test` command. ```groovy Jenkinsfile theme={null} pipeline { agent any tools {nodejs "Node 20"} environment { CHECKLY_API_KEY = credentials('checkly-api-key') CHECKLY_ACCOUNT_ID = credentials('checkly-account-id') CHECKLY_TEST_ENVIRONMENT='production' } stages { stage('Deploy') { steps { echo 'Deploying....' } } stage('Dependencies') { steps { sh 'npm ci' } } stage('checkly test') { steps { sh 'npx checkly test' } } stage('checkly deploy') { when { branch "main" } steps { sh 'npx checkly deploy --force' } } } } ``` # CI/CD Source: https://www.checklyhq.com/docs/integrations/ci-cd/overview Core to our Monitoring as Code approach is running your checks from your CI/CD pipeline and use them as E2E tests. Regardless of the provider or platform you use, integrating Checkly into your CI/CD pipeline boils down to **four basic steps**: 1. Store your checks as code and `git push`. Preferably store your checks alongside your application code. Just like unit tests. 2. Wait for your application to be deployed. This can be done through hooks, lifecycle events or wait scripts. 3. Run your checks using `npx checkly test` and pass in any variables like an `ENVIRONMENT_URL`. 4. If the checks pass successfully, run `npx checkly deploy`. With this approach, you can validate your application and infrastructure before deploying to production, staging or any other environments. Simultaneously , you can life cycle (create/update/delete) your checks from your code base as part of your CI/CD workflow. ## CI/CD Integration with the Checkly CLI The preferred and most flexible way to integrate Checkly with your CI/CD is through the [Checkly CLI](/docs/cli/overview). Run Checkly test sessions from GitHub Actions and report results in pull requests. Run the Checkly CLI from GitLab CI pipelines, using separate e2e-test and deploy jobs. Run the Checkly CLI from a Jenkins pipeline using a Jenkinsfile. ## CI/CD Integration using vendor webhooks * [Vercel](/docs/integrations/ci-cd/vercel/overview) * [GitHub deployment hooks](/docs/integrations/ci-cd/github/deployments) ## CI/CD Basics We've optimized the [Checkly CLI](/docs/cli/overview) to work in any CI/CD workflow. Here are the basics you need to know that will come in handy when adapting the examples we give you to your own, specific setup. 1. For **authentication**, make sure to set the `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` parameters as environment variables in your CI/CD platform. 2. Set the **reporter** you want to use for the `test` command using the `--reporter` flag, i.e. `--reporter=dot`. 3. Use `npx checkly test` to validate your checks before deploying. Test results, traces, and videos are available in Checkly as a **test session**. 4. Use the `--force` flag on the `deploy` and / or `destroy` commands to skip the normal confirmation steps. The CLI will attempt to parse `git` specific information from the environment to display in the test session as metadata. You can also set these data items specifically by using environment variables. | Item | Auto | Variable | Description | | ------------------ | ----- | ------------------------------------------------------ | ------------------------------------------- | | **Repository** | false | `repoUrl` in `checkly.config.ts` or `CHECKLY_REPO_URL` | The URL of your repo on GitHub, GitLab etc. | | **Commit hash** | true | `CHECKLY_REPO_SHA` | The SHA of the commit. | | **Branch** | true | `CHECKLY_REPO_BRANCH` | The branch name. | | **Commit owner** | true | `CHECKLY_REPO_COMMIT_OWNER` | The committer's name or email. | | **Commit message** | true | `CHECKLY_REPO_COMMIT_MESSAGE` | The commit message. | | **Environment** | false | `CHECKLY_TEST_ENVIRONMENT` | The environment name, e.g. "staging" | # Vercel Deployment Protection Source: https://www.checklyhq.com/docs/integrations/ci-cd/vercel/deployment-protection Let Checkly's browser checks reach protected Vercel preview and production deployments by passing a Protection Bypass for Automation secret in your tests. In some cases, you will need to provide authentication credentials to your Vercel deployment in order to run your browser checks. Vercel calls this [**Deployment Protection**](https://vercel.com/docs/security/deployment-protection). To make optimal use of Checkly's Vercel integration, Checkly will need access to Preview deployments and Production deployments, so we can validate your preview branches before they reach production and continuously monitor your production deployments. Vercel has a few different ways to authenticate your deployments. Each work a bit different. We will go over each of them below. ## Bypass for Automation You can fully bypass any deployment protection using Vercel's [Protection Bypass for Automation feature](https://vercel.com/docs/security/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation). With this feature, you can provide a secret in an HTTP header or query parameter to bypass any deployment protection. ### How to make this work with Checkly? For any Playwright-powered check, like Browser checks and Multistep checks, you can add the secret to the required HTTP header using the `test.use()` method. ```ts browser-check.spec.ts theme={null} test.use({ extraHTTPHeaders: { 'x-vercel-protection-bypass': process.env.VERCEL_BYPASS_TOKEN } }) ``` Alternatively, when using the `request` API, you can add the header to the request. ```ts api-request.spec.ts theme={null} await request.get('https://my-production-url.vercel.app/', { 'x-vercel-protection-bypass': process.env.VERCEL_BYPASS_TOKEN }) ``` For an API check you can do something similar 1. Just add a `x-vercel-protection-bypass` header to the request with the token 2. Add a query parameter to the request with the token. 3. Or use a setup script to manipulate the request before it is sent. ```bash setup-script.js theme={null} request.headers['x-vercel-protection-bypass'] = process.env.VERCEL_BYPASS_TOKEN ``` ## Vercel Authentication Vercel has a few different ways to authenticate your deployments. Each work a bit different and require a different approach to make them work with Checkly. > Vercel enables **Standard Protection** by default on all new deployments. ### Standard Protection [Standard Protection](https://vercel.com/docs/concepts/deployments/deployment-protection#vercel-authentication) works by prompting the user to log in to Vercel with their normal Vercel user before they can access the deployment. You can tweak this setting in the Vercel dashboard under *Settings > General > Deployment Protection*. vercel deployment protection

#### How to make this work with Checkly? 1. You can completely disable Standard Protection in the Vercel dashboard. This will make your deployments publicly accessible. 2. You can add some extra code to a browser check to actually log in to Vercel. Here is an example of how that would work if you are authenticating using GitHub. Note: if you have 2FA enabled on your GitHub account, this will not work. ```typescript visit-protected.spec.ts theme={null} import { expect, test } from '@playwright/test' test('visit protected deployment', async ({ page }) => { const response = await page.goto(process.env.ENVIRONMENT_URL || 'https://my-production-url.vercel.app/') // Auth dialog is a pop up const popupPromise = page.waitForEvent('popup'); await page.getByText('Continue with GitHub').click() const popup = await popupPromise; await popup.waitForLoadState(); await popup.locator('input[name="login"]').type(process.env.GITHUB_USER) await popup.locator('input[name="password"]').type(process.env.GITHUB_PASSWORD) await popup.getByText('Sign in', {exact: true}).click() await page.screenshot({ path: 'screenshot.jpg' }) expect(response.status(), 'should respond with correct status code').toBeLessThan(400) }) ``` ```javascript visit-protected.spec.js theme={null} const { expect, test } = require('@playwright/test') test('visit protected deployment', async ({ page }) => { const response = await page.goto(process.env.ENVIRONMENT_URL || 'https://my-production-url.vercel.app/') // Auth dialog is a pop up const popupPromise = page.waitForEvent('popup'); await page.getByText('Continue with GitHub').click() const popup = await popupPromise; await popup.waitForLoadState(); await popup.locator('input[name="login"]').type(process.env.GITHUB_USER) await popup.locator('input[name="password"]').type(process.env.GITHUB_PASSWORD) await popup.getByText('Sign in', { exact: true }).click() await page.screenshot({ path: 'screenshot.jpg' }) expect(response.status(), 'should respond with correct status code').toBeLessThan(400) }) ``` You will have to supply your GitHub username and password as environment variables (e.g. `GITHUB_USER` and `GITHUB_PASSWORD` in the code example) to your Checkly account or as part of the [browser check's environment variables](/docs/platform/variables). ### Only Preview Deployments [Only Preview Deployments](https://vercel.com/docs/concepts/deployments/deployment-protection#vercel-authentication) is the same as Standard Protection, but only applies to Preview deployments. This means that we need to tweak the above code a bit to check if we are targeting a Preview deployment or not. #### How to make this work with Checkly? We just need to check if the automatically injected `ENVIRONMENT_URL` environment variable is set. If it is, we know we are checking a Preview deployment and we can run the authentication code. If it is not set, we just proceed as normal. ```typescript visit-protected.spec.ts theme={null} import { expect, test } from '@playwright/test' test('visit protected deployment', async ({ page }) => { const response = await page.goto(process.env.ENVIRONMENT_URL || 'https://my-production-url.vercel.app/') if (!!process.env.ENVIRONMENT_URL) { // Auth dialog is a pop up const popupPromise = page.waitForEvent('popup'); await page.getByText('Continue with GitHub').click() const popup = await popupPromise; await popup.waitForLoadState(); await popup.locator('input[name="login"]').type(process.env.GITHUB_USER) await popup.locator('input[name="password"]').type(process.env.GITHUB_PASSWORD) await popup.getByText('Sign in', {exact: true}).click() } await page.screenshot({ path: 'screenshot.jpg' }) expect(response.status(), 'should respond with correct status code').toBeLessThan(400) }) ``` ```javascript visit-protected.spec.js theme={null} const { expect, test } = require('@playwright/test') test('visit protected deployment', async ({ page }) => { const response = await page.goto(process.env.ENVIRONMENT_URL || 'https://my-production-url.vercel.app/') // If we if (!!process.env.ENVIRONMENT_URL) { // Auth dialog is a pop up const popupPromise = page.waitForEvent('popup'); await page.getByText('Continue with GitHub').click() const popup = await popupPromise; await popup.waitForLoadState(); await popup.locator('input[name="login"]').type(process.env.GITHUB_USER) await popup.locator('input[name="password"]').type(process.env.GITHUB_PASSWORD) await popup.getByText('Sign in', { exact: true }).click() } await page.screenshot({ path: 'screenshot.jpg' }) expect(response.status(), 'should respond with correct status code').toBeLessThan(400) }) ``` If you have 2FA enabled on your GitHub or other Vercel authentication methods, the above example will not work. You can explore bypassing 2FA using the `otpauth` library. [Check our blog post for more info](https://www.checklyhq.com/blog/how-to-bypass-totp-based-2fa-login-flows-with-playwright/). ## Password Protection Vercel's [Password Protection](https://vercel.com/docs/concepts/deployments/deployment-protection#password-protection) is a paid addon that allows you to set a dedicated password for your deployments and specifically bypass protection for E2E tests / automation. You can accomplish this using the resources referenced in the docs for [browser checks](/docs/detect/synthetic-monitoring/browser-checks/examples#username--password-login) or [API requests](https://www.checklyhq.com/docs/detect/synthetic-monitoring/api-checks/examples#dismiss-password-protection-prompt-on-vercel-deployment). * [IP Allowlisting](https://vercel.com/docs/concepts/deployments/deployment-protection#ip-allowlisting) # Managing Users and Plans with Vercel Source: https://www.checklyhq.com/docs/integrations/ci-cd/vercel/managing-plan Manage Checkly users, plans, and billing through Vercel's native integration, adding team members and adjusting your plan from the Vercel dashboard. The Vercel-[Checkly native integration](https://vercel.com/marketplace/checkly) moves the user and plan management of your Checkly account to Vercel. You have access to the same Hobby and Team plan as when subscribing directly on Checkly, but the management process is different. If you are new to Vercel's native integrations you can read more about them [here](https://vercel.com/docs/integrations#native-integrations). ### User management If you are using the Vercel native integration, your Checkly users will be managed through Vercel. Members of your Vercel team can all access Checkly up to the seat limit on your selected plan. See our [pricing page](https://www.checklyhq.com/pricing/) for details on the number of seats available on your plan. To add a new user to your Checkly account they need to be a member of your Vercel team. The user then only needs to click `Open in Checkly` to be added to your Checkly account. ### Usage and billing You can view and adjust your selected plan from the integration product page in Vercel. Go to the [your Checkly account billing page](https://app.checklyhq.com/settings/account/billing) and click `Visit Vercel`, select the installed product, then select `Settings`. Under `Change configuration` you can change your current plan, or change the number of pre-purchased check runs on a team plan subscription. You can always view your recent usage under the `Usage` tab, found below `Settings` in the sidebar. Checkly invoices will be sent to our Vercel account, and can be viewed under `Settings` and `Invoices` in Vercel. If you have an active subscription to a paid plan, you cannot delete your integration before you have downgraded your plan to a free one. This is to ensure that you have control over your installation and can utilize features and capabilities you have paid for until the end of the subscription. If you need to delete the integration earlier, contact [support@checklyhq.com](mailto:support@checklyhq.com) for assistance. # Integrating Checkly in Vercel Source: https://www.checklyhq.com/docs/integrations/ci-cd/vercel/overview Integrate Checkly with Vercel using the Marketplace and Connected integrations to manage your account and monitor preview and production deployments. If you are using Vercel to develop, preview, and ship your application, you can natively integrate with Checkly by installing [the official integration](https://vercel.com/marketplace/checkly) from the Vercel Marketplace. Checkly offers two integrations for Vercel users: * Marketplace Integration, **for managing your Checkly account and users**. With the native integration, user access and billing is managed through Vercel. If you are new to Vercel's native integrations you can read more about them on the [Vercel documentation page](https://vercel.com/docs/integrations#native-integrations). * Connected Integration, **to monitor your preview and production deployments**. The connected integration can be installed stand-alone or on top of the native integration and helps you by: * Automatically creating a pre-scripted browser check to catch any errors and failed requests as your web page loads. * Enabling you to run checks against preview and production deployments on Vercel. Using Vercel deployment protection? Read [this section](/docs/integrations/ci-cd/vercel/deployment-protection) on how to make this work with Checkly. ## Installing the Marketplace Integration To install Checkly's managed Vercel integration, navigate to the [integration's marketplace listing](https://vercel.com/integrations/checkly) and click `Install`. 1. Select `Create new account` in the installation guide and click `Next` to accept the terms and conditions. 2. Select your plan type. You can read about our pricing plans [here](https://www.checklyhq.com/pricing/). 3. Name your account and review your plan selection. You can change your account name later. 4. Press `Create` to continue to the integration settings page. Select `Getting started` to go to the setup guide in Vercel. You can also click `Open in Checkly` to go through the Checkly + Vercel onboarding and get you started with an example project showcasing how to use Monitoring as Code together with Vercel. You can find the [example NextJS project on GitHub](https://github.com/checkly/nextjs-checkly-starter-template) which includes a guide on how to use Monitoring as Code with a NextJS project. ## Installing the connected integration This integration has some [known limitations](/docs/integrations/ci-cd/vercel/overview#limitations). We recommend using the [Checkly CLI](/docs/cli/overview) in your CI/CD pipeline instead. 1. To install Checkly's connected Vercel integration, navigate to [Integrations](https://app.checklyhq.com/settings/account/integrations), under your account's dropdown menu and click the `Vercel marketplace` button, or go directly to the [integration listing on the Vercel marketplace](https://vercel.com/marketplace/checkly). 2. On the marketplace page for Checkly, click `Connect Account`. 3. Next, follow the installation wizard to grant the integration access to the right Vercel scope and projects. 4. You can choose to map your Vercel projects to existing checks on your Checkly account, to have them run on production and/or preview deployments. 5. Additionally, you can have new checks automatically generated for existing Vercel projects, and set them up to automatically run on preview and/or production deployments. ## Managing the connected integration If you already have the connected Vercel integration set up, you might still want to connect new checks or groups. The procedure is the same for both: edit the check or group and select `Vercel` in the `Testing` tab, then select `Link Vercel project` and the project you want to link. linking existing check

Once the project and the check/group have been linked, you are able to specify whether a new preview or production deployment should trigger a new execution. You will also have the chance to choose from which location this check will run. deployment trigger settings

Should you wish to unlink the Vercel project, simply click `Unlink this project`. ### Targeting Preview deployments with `ENVIRONMENT_URL` and `ENVIRONMENT_NAME` On each deployment triggered by Vercel, we inject the Checkly runtime with two variables: 1. `ENVIRONMENT_NAME`: This name is `preview` for preview deployments and `production` for production deployments. 2. `ENVIRONMENT_URL`: For **Preview deployments** this is the preview URL alias, e.g. `https://ecommerce-1wabp0sqy-acme.vercel.app`, for **Production deployments** it is the equivalent production URL alias. 3. `DEPLOYMENT_ID`: The unique deployment ID generated by Vercel. You can access all these variables with the familiar `process.ENV` syntax. ### Browser checks This means you can write a Browser check using Playwright Test or Playwright that automatically targets the Preview deploy whenever this URL is available, but otherwise just defaults to the stable **production URL**. Here is a full example that we use ourselves to monitor checklyhq.com which is actually also deployed to Vercel. ```typescript vercel.spec.ts theme={null} import { expect, test } from '@playwright/test' test('assert response status of page', async ({ page }) => { const targetUrl = process.env.ENVIRONMENT_URL || 'https://www.checklyhq.com' const response = await page.goto(targetUrl) expect(response.status()).toBeLessThan(400) }) ``` ```javascript vercel.spec.js theme={null} const { expect, test } = require('@playwright/test') test('assert response status of page', async ({ page }) => { const targetUrl = process.env.ENVIRONMENT_URL || 'https://www.checklyhq.com' const response = await page.goto(targetUrl) expect(response.status()).toBeLessThan(400) }) ``` This way we are setting the `targetUrl` variable to either the `ENVIRONMENT_URL` or just our main production URL. Whenever a **Preview** deploy happens on Vercel, this check gets called and runs the script against the preview environment. This check also runs on a 5 minute schedule, and checks our production environment. This way, we kill two birds with one stone and don't need separate checks for separate environments. ### API checks With API checks, we automatically replace the hostname part of your request URL with the host in the environment URL. For example: * Your configured URL: [https://api.acme.com/v1/customers?page=1](https://api.acme.com/v1/customers?page=1) * Environment URL: [https://now.customer-api.qis6va2z7.now.sh](https://now.customer-api.qis6va2z7.now.sh) * Replaced URL: [https://now.customer-api.qis6va2z7.now.sh/v1/customers?page=1](https://now.customer-api.qis6va2z7.now.sh/v1/customers?page=1) Notice we only replace the host part, not the URL path or any query parameters. Just like for browser checks, the check will run on deploy against our preview environment, while also still running on a schedule against production. ### How Checkly checks maps to Vercel checks Checkly integrates deeply with the Vercel [Checks functionality](https://vercel.com/docs/concepts/deployments/checks) API to link Checks in both tools. Vercel uses a slightly different way in representing checks than Checkly does, specifically splitting individual check results into either: 1. **Reliability checks** 2. **Performance checks** This is how a Checkly check maps to a Vercel check: * **API checks** are always mapped 1:1 and marked as a **Reliability** check. * **Browser checks** are split into a **Performance** and **Reliability** check, where the Performance part is populated with Web Vitals data\*. * **Groups** of checks are "unfolded" in the Vercel user interface and all checks are shown individually. \* *Web Vitals are only recorded for [Playwright-based Browser checks](/docs/detect/synthetic-monitoring/browser-checks/performance-metrics#web-performance-vitals)* ### Blocking Vercel deployments You can block your Vercel deployments if any of your checks fail or if the web vitals degrade: 1. **Block my deployment when a check fails** does what it says on the tin and only applies to Vercel **Reliability checks**. 2. **Block my deployment when Web Vitals degrade** is a bit special and only applies to Vercel **Performance checks** for our Browser checks. This table will help you determine how blocking checks works. Note — again — that **Browser checks**, can fail a Vercel deployment in two ways because the Reliability part and Performance part are evaluated separately. | Checkly | Vercel | Blocking heuristic | | ------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | API Check | Reliability Check | Blocks deployment when it fails due to > 399 HTTP response codes, assertions and other API check specific features. | | Browser Check | Reliability Check | Blocks deployment when it fails due to > 399 HTTP response codes on the main HTML document, syntax errors in the script, or assertions using the `expect` or `assert` libraries in the browser check script. | | | Performance Check | Blocks deployment when a degradation is registered in the aggregate **Virtual Experience Score** based on Web Vitals. Read below for more details. | > Reliability and performance checks run against the [automatic deployment URL](https://vercel.com/docs/concepts/deployments/automatic-urls) Vercel generates. In case of failures or degradations, what *actually* gets blocked is the deployment to all other URLs, e.g. branch URLs and user-assigned domains. ### Virtual Experience Score & Web Vitals Together with the team at Vercel, we developed the **Virtual Experience Score** (VES) that gives you one simple KPI to track. The Virtual Experience Score does three things: 1. It aggregates your Web Vitals metrics into one number from 0 to 100, just like you might have seen with [Lighthouse scoring](https://web.dev/performance-scoring/). 2. It assigns a "weight" to each score, marking its impact on your UX. 3. It chops up the score into the top 10% (p10), the top 50% (median) and the rest. In the table below you can see the exact numbers and their weight in the Virtual Experience Score. | Web Vitals metrics | p10 / 90% | median / 50% | weight | | ------------------ | --------- | ------------ | ------ | | FCP | 934ms | 1600ms | 0.20 | | LCP | 1200ms | 2400ms | 0.35 | | TBT | 150ms | 350ms | 0.30 | | CLS | 0.1 | 0.25 | 0.15 | So, what does this mean for the blocking heuristics? It means that we will mark a check as **blocking** when: 1. A check reports web vitals, and... 2. The VES has degraded in relation to the last available **Production** deploy. Degraded means they passed the range threshold, i.e from **p10** to **median**. For more info on the **Virtual Experience Score** [check the documentation on the Vercel site](https://vercel.com/docs/concepts/analytics/web-vitals#virtual-experience-score). ### Skipping Performance checks In some cases, Checkly will completely skip performance checks. You will see the "skipped" status in your Vercel deployment overview. Checkly skips performance checks when the domain of the visited URL in the script does not match the domain of the **deployment URL**. In 9 out of 10 cases this should be the URL for your Preview and Production deployments. ### Vercel-linked test session results Any results triggered by a deployment can be viewed on the test sessions page in Checkly. Runs triggered by the Vercel integration have the Vercel logo next to the user name. cicd triggered check results tab

On Vercel, you will also see a breakdown of checks that were executed on a given deployment, together with a breakdown of various key web vitals. vercel checks vitals

### Limitations The Vercel connected integration lacks support for several features: * Client certificates are not applied. * OpenTelemetry integration headers are not applied. * Private locations are not available. As an alternative, we recommend using the [Checkly CLI](/docs/cli/overview) in your CI/CD pipeline. The CLI is much more powerful and fully supported. # Pulumi Best Practices Source: https://www.checklyhq.com/docs/integrations/iac/pulumi/best-practices Best practices for managing monitoring as code with the Checkly Pulumi provider This guide covers best practices for effectively managing your Checkly monitoring infrastructure using the Pulumi provider, ensuring maintainable, secure, and scalable monitoring as code. ## Resource Management ### Naming Conventions **Consistent Naming** * Use descriptive, lowercase names with hyphens * Include environment prefixes (dev, staging, prod) * Follow team-established conventions * Avoid generic names like "check1" or "test" **Organized Tagging** * Use consistent tag patterns across resources * Include environment, team, and purpose tags * Enable easy filtering and reporting * Document your tagging strategy **Good Examples:** ```javascript theme={null} // Good naming examples new checkly.Check('prod-api-health-check', { name: 'Production API Health Check', tags: ['prod', 'api', 'health', 'team-backend'], }) new checkly.Check('staging-login-flow-check', { name: 'Staging Login Flow Check', tags: ['staging', 'browser', 'auth', 'team-frontend'], }) ``` **Avoid:** ```javascript theme={null} // Avoid these naming patterns new checkly.Check('check1', { name: 'Test' }) new checkly.Check('my_check', { name: 'My Check' }) ``` ### Resource Organization Organize your Pulumi code into logical modules and use check groups to maintain clear relationships between related resources. ```javascript theme={null} // Organize by service/domain const apiChecks = require('./checks/api') const browserChecks = require('./checks/browser') const alertChannels = require('./alerts') // Create service-specific groups const apiGroup = new checkly.CheckGroup('api-monitoring', { name: 'API Monitoring', tags: ['api', 'monitoring'], }) // Add related checks to groups apiChecks.forEach(check => { new checkly.Check(check.name, { ...check, groupId: apiGroup.id, }) }) ``` ## Security Best Practices ### Secret Management Never commit API keys, passwords, or other sensitive information directly in your Pulumi code or version control. **Use Environment Variables:** ```javascript theme={null} // Good: Use environment variables new checkly.Check('authenticated-api', { request: { headers: { 'Authorization': `Bearer ${process.env.API_TOKEN}`, }, }, }) ``` **Use Pulumi Configuration:** ```bash theme={null} # Set secrets in Pulumi config pulumi config set checkly:apiKey your_api_key --secret pulumi config set checkly:accountId your_account_id ``` **Access in Code:** ```javascript theme={null} const config = new pulumi.Config() const apiKey = config.requireSecret('checkly:apiKey') const accountId = config.require('checkly:accountId') ``` ### Access Control **Secure API Keys** * Use dedicated API keys for Pulumi * Rotate keys regularly * Limit key permissions when possible * Monitor key usage **Team Permissions** * Use Pulumi organizations for team collaboration * Implement role-based access control * Review access permissions regularly * Use separate stacks for different environments ## Environment Management ### Stack Strategy Use separate Pulumi stacks for different environments to maintain isolation and enable environment-specific configurations. ```bash theme={null} # Create environment-specific stacks pulumi stack init dev pulumi stack init staging pulumi stack init prod ``` **Environment-Specific Configuration:** ```javascript theme={null} const config = new pulumi.Config() const environment = config.require('environment') const envConfig = { dev: { frequency: 30, locations: ['eu-west-1'], alertChannels: ['dev-alerts'], }, staging: { frequency: 15, locations: ['eu-west-1', 'us-west-2'], alertChannels: ['staging-alerts'], }, prod: { frequency: 5, locations: ['eu-west-1', 'us-west-2', 'ap-southeast-1'], alertChannels: ['prod-alerts', 'pager-duty'], }, } const currentConfig = envConfig[environment] ``` ### Configuration Management Use Pulumi's configuration system to manage environment-specific settings and avoid hardcoding values. ```bash theme={null} # Set environment-specific configs pulumi config set environment dev --stack dev pulumi config set environment staging --stack staging pulumi config set environment prod --stack prod # Set environment-specific URLs pulumi config set apiUrl https://dev-api.example.com --stack dev pulumi config set apiUrl https://staging-api.example.com --stack staging pulumi config set apiUrl https://api.example.com --stack prod ``` ## Monitoring Strategy ### Check Frequency Optimization **High Frequency** * 1-5 minute intervals for critical APIs * Multiple locations for redundancy * Immediate alerting on failures * Comprehensive assertions **Lower Frequency** * 10-30 minute intervals for non-critical services * Single location monitoring * Delayed alerting * Basic health checks ### Location Strategy ```javascript theme={null} // Critical service - multiple locations new checkly.Check('critical-payment-api', { frequency: 1, locations: ['eu-west-1', 'us-west-2', 'ap-southeast-1'], tags: ['critical', 'payment'], }) // Non-critical service - single location new checkly.Check('admin-dashboard', { frequency: 30, locations: ['eu-west-1'], tags: ['admin', 'non-critical'], }) ``` ### Assertion Best Practices **Comprehensive API Checks:** ```javascript theme={null} new checkly.Check('comprehensive-api-check', { request: { method: 'GET', url: 'https://api.example.com/health', assertions: [ // Status code check { source: 'STATUS_CODE', comparison: 'EQUALS', target: '200', }, // Response time check { source: 'RESPONSE_TIME', comparison: 'LESS_THAN', target: '1000', }, // JSON body validation { source: 'JSON_BODY', property: '$.status', comparison: 'EQUALS', target: 'healthy', }, // Header validation { source: 'HEADERS', property: 'content-type', comparison: 'CONTAINS', target: 'application/json', }, ], }, }) ``` ## Code Organization ### Modular Structure Organize your Pulumi code into logical modules to improve maintainability and reusability. **Recommended Project Structure:** ``` my-checkly-monitoring/ ├── index.js # Main entry point ├── checks/ │ ├── api.js # API check definitions │ ├── browser.js # Browser check definitions │ └── groups.js # Check group definitions ├── alerts/ │ └── channels.js # Alert channel definitions ├── config/ │ └── environments.js # Environment configurations └── utils/ └── helpers.js # Helper functions ``` **Module Example:** ```javascript theme={null} // checks/api.js const checkly = require('@checkly/pulumi') function createApiChecks(environment) { return [ { name: 'health-check', url: `https://${environment}-api.example.com/health`, frequency: environment === 'prod' ? 5 : 30, }, { name: 'metrics-check', url: `https://${environment}-api.example.com/metrics`, frequency: environment === 'prod' ? 10 : 60, }, ] } module.exports = { createApiChecks } ``` ### Reusable Functions ```javascript theme={null} // utils/helpers.js function createStandardApiCheck(name, url, options = {}) { return { name, activated: true, frequency: options.frequency || 10, type: 'API', locations: options.locations || ['eu-west-1'], tags: options.tags || [], request: { method: 'GET', url, assertions: [ { source: 'STATUS_CODE', comparison: 'EQUALS', target: '200', }, ...(options.assertions || []), ], }, useGlobalAlertSettings: true, } } module.exports = { createStandardApiCheck } ``` ## CI/CD Integration ### Automated Deployments **GitHub Workflow** * Deploy on push to main branch * Preview changes on pull requests * Use Pulumi GitHub Actions * Secure secret management **Other Platforms** * GitLab CI/CD integration * Jenkins pipeline support * Azure DevOps integration * CircleCI configuration **GitHub Actions Example:** ```yaml theme={null} name: Deploy Monitoring Infrastructure on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Setup Node.js uses: actions/setup-node@v3 with: node-version: '18' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install Dependencies run: npm install - name: Deploy to Production run: pulumi up --yes --stack prod env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} ``` ## Testing and Validation ### Preview Changes Always use `pulumi preview` before deploying to understand what changes will be made to your infrastructure. ```bash theme={null} # Preview changes before deployment pulumi preview --stack prod # Preview specific changes pulumi preview --stack prod --show-replacement-steps ``` ### Validation Strategies * Validate environment variables are set * Check API key permissions * Verify account ID format * Test connectivity to Checkly API * Verify check configurations * Test alert channel connectivity * Validate check group assignments * Confirm location availability * Monitor initial check runs * Verify alert channel notifications * Check resource relationships * Validate performance metrics ## Performance Considerations ### Resource Limits Be mindful of Checkly's resource limits and pricing when creating large numbers of checks or high-frequency monitoring. **Optimization Strategies:** * Use appropriate check frequencies * Group related checks to reduce overhead * Monitor resource usage and costs * Implement check deactivation for non-critical services ### Scalability ```javascript theme={null} // Use loops for similar checks const endpoints = [ '/health', '/metrics', '/status', '/ready', ] endpoints.forEach(endpoint => { new checkly.Check(`api-${endpoint.replace('/', '')}`, { name: `API ${endpoint} Check`, request: { method: 'GET', url: `https://api.example.com${endpoint}`, }, // ... other configuration }) }) ``` ## Maintenance and Updates ### Regular Reviews Schedule regular reviews of your monitoring infrastructure to ensure it remains effective and up-to-date. **Review Checklist:** * [ ] Check for unused or obsolete resources * [ ] Review and update check frequencies * [ ] Validate alert channel configurations * [ ] Update check assertions and thresholds * [ ] Review performance and cost metrics * [ ] Update documentation and runbooks ### Documentation Maintain comprehensive documentation of your monitoring infrastructure, including runbooks, troubleshooting guides, and configuration details. **Documentation Requirements:** * Resource naming conventions * Environment configurations * Alert escalation procedures * Troubleshooting guides * Change management procedures ## Common Pitfalls to Avoid **Problem**: Manual changes in Checkly UI create drift with Pulumi state **Solution**: Always manage resources through Pulumi or document manual changes **Problem**: Committing secrets to version control **Solution**: Use environment variables and Pulumi's secret management **Problem**: Too many checks or too frequent monitoring **Solution**: Focus on critical paths and optimize check frequencies **Problem**: Unclear or inconsistent resource names **Solution**: Establish and follow naming conventions Remember that monitoring infrastructure is critical to your application's reliability. Always test changes in development environments before deploying to production. # CI/CD Integration Source: https://www.checklyhq.com/docs/integrations/iac/pulumi/ci-cd Integrate Pulumi with CI/CD pipelines for automated monitoring infrastructure deployments This guide shows you how to integrate the Checkly Pulumi provider with various CI/CD platforms to automate your monitoring infrastructure deployments. ## Overview CI/CD integration enables you to automatically deploy monitoring changes when code is pushed, ensuring your monitoring infrastructure stays in sync with your application deployments. **Continuous Deployment** * Deploy monitoring changes automatically * Reduce manual configuration errors * Ensure infrastructure consistency * Enable rapid iteration **Multi-Environment Support** * Separate stacks for each environment * Environment-specific configurations * Controlled promotion between environments * Consistent monitoring across environments **Secure Deployments** * Secure secret management * Role-based access control * Audit trails for changes * Compliance with security policies **Team Workflows** * Code review for monitoring changes * Pull request-based deployments * Collaborative infrastructure management * Standardized deployment processes ## GitHub Actions ### Basic Workflow ```yaml .github/workflows/deploy.yml theme={null} name: Deploy Monitoring Infrastructure on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install dependencies run: npm ci - name: Deploy to production run: pulumi up --yes --stack prod env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} ``` ```yaml .github/workflows/preview.yml theme={null} name: Preview Monitoring Changes on: pull_request: branches: [main] jobs: preview: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install dependencies run: npm ci - name: Preview changes run: pulumi preview --stack prod env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} ``` ### Multi-Environment Workflow ```yaml .github/workflows/multi-env.yml theme={null} name: Deploy Monitoring Infrastructure on: push: branches: [main, staging] jobs: deploy: runs-on: ubuntu-latest strategy: matrix: environment: [staging, prod] include: - environment: staging branch: staging stack: staging - environment: prod branch: main stack: prod if: github.ref == "refs/heads/${{ matrix.branch }}" steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install dependencies run: npm ci - name: Deploy to ${{ matrix.environment }} run: pulumi up --yes --stack ${{ matrix.stack }} env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} ``` ### Advanced Workflow with Testing ```yaml .github/workflows/pipeline.yml theme={null} name: Monitoring Infrastructure Pipeline on: push: branches: [main, staging] pull_request: branches: [main] jobs: test: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install dependencies run: npm ci - name: Run tests run: npm test - name: Lint code run: npm run lint preview: needs: test runs-on: ubuntu-latest if: github.event_name == 'pull_request' steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install dependencies run: npm ci - name: Preview changes run: pulumi preview --stack staging env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} deploy-staging: needs: test runs-on: ubuntu-latest if: github.ref == 'refs/heads/staging' environment: staging steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install dependencies run: npm ci - name: Deploy to staging run: pulumi up --yes --stack staging env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} deploy-production: needs: [test, deploy-staging] runs-on: ubuntu-latest if: github.ref == 'refs/heads/main' environment: production steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install Pulumi uses: pulumi/setup-pulumi@v2 - name: Install dependencies run: npm ci - name: Deploy to production run: pulumi up --yes --stack prod env: CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} ``` ## GitLab CI/CD ### Basic Pipeline ```yaml .gitlab-ci.yml theme={null} stages: - test - preview - deploy variables: NODE_VERSION: "18" test: stage: test image: node:${NODE_VERSION} script: - npm ci - npm test - npm run lint only: - merge_requests - main - staging preview: stage: preview image: node:${NODE_VERSION} before_script: - curl -fsSL https://get.pulumi.com | sh - export PATH=$PATH:$HOME/.pulumi/bin - npm ci script: - pulumi preview --stack staging only: - merge_requests environment: name: staging url: https://app.checklyhq.com deploy-staging: stage: deploy image: node:${NODE_VERSION} before_script: - curl -fsSL https://get.pulumi.com | sh - export PATH=$PATH:$HOME/.pulumi/bin - npm ci script: - pulumi up --yes --stack staging only: - staging environment: name: staging url: https://app.checklyhq.com variables: CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID CHECKLY_API_KEY: $CHECKLY_API_KEY PULUMI_ACCESS_TOKEN: $PULUMI_ACCESS_TOKEN deploy-production: stage: deploy image: node:${NODE_VERSION} before_script: - curl -fsSL https://get.pulumi.com | sh - export PATH=$PATH:$HOME/.pulumi/bin - npm ci script: - pulumi up --yes --stack prod only: - main environment: name: production url: https://app.checklyhq.com variables: CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID CHECKLY_API_KEY: $CHECKLY_API_KEY PULUMI_ACCESS_TOKEN: $PULUMI_ACCESS_TOKEN ``` ## Jenkins ### Jenkinsfile ```groovy Jenkinsfile theme={null} pipeline { agent any environment { NODE_VERSION = '18' PULUMI_VERSION = '3.100.0' } stages { stage('Checkout') { steps { checkout scm } } stage('Setup') { steps { sh ''' # Install Node.js curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | sudo -E bash - sudo apt-get install -y nodejs # Install Pulumi curl -fsSL https://get.pulumi.com | sh export PATH=$PATH:$HOME/.pulumi/bin # Install dependencies npm ci ''' } } stage('Test') { steps { sh ''' npm test npm run lint ''' } } stage('Preview') { when { changeRequest() } steps { sh ''' export PATH=$PATH:$HOME/.pulumi/bin pulumi preview --stack staging ''' } } stage('Deploy Staging') { when { branch 'staging' } steps { sh ''' export PATH=$PATH:$HOME/.pulumi/bin pulumi up --yes --stack staging ''' } } stage('Deploy Production') { when { branch 'main' } steps { sh ''' export PATH=$PATH:$HOME/.pulumi/bin pulumi up --yes --stack prod ''' } } } post { always { cleanWs() } } } ``` ## Azure DevOps ### azure-pipelines.yml ```yaml azure-pipelines.yml theme={null} trigger: branches: include: - main - staging pr: branches: include: - main pool: vmImage: 'ubuntu-latest' variables: nodeVersion: '18.x' pulumiVersion: '3.100.0' stages: - stage: Test displayName: 'Test and Validate' jobs: - job: Test displayName: 'Run Tests' steps: - task: NodeTool@0 inputs: versionSpec: $(nodeVersion) displayName: 'Install Node.js' - script: | npm ci displayName: 'Install dependencies' - script: | npm test displayName: 'Run tests' - script: | npm run lint displayName: 'Run linting' - stage: Preview displayName: 'Preview Changes' dependsOn: Test condition: eq(variables['Build.Reason'], 'PullRequest') jobs: - job: Preview displayName: 'Preview Infrastructure Changes' steps: - task: NodeTool@0 inputs: versionSpec: $(nodeVersion) displayName: 'Install Node.js' - script: | curl -fsSL https://get.pulumi.com | sh export PATH=$PATH:$HOME/.pulumi/bin npm ci displayName: 'Setup Pulumi and install dependencies' - script: | export PATH=$PATH:$HOME/.pulumi/bin pulumi preview --stack staging displayName: 'Preview changes' env: CHECKLY_ACCOUNT_ID: $(CHECKLY_ACCOUNT_ID) CHECKLY_API_KEY: $(CHECKLY_API_KEY) PULUMI_ACCESS_TOKEN: $(PULUMI_ACCESS_TOKEN) - stage: DeployStaging displayName: 'Deploy to Staging' dependsOn: Test condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/staging')) jobs: - deployment: DeployStaging displayName: 'Deploy to Staging' environment: staging strategy: runOnce: deploy: steps: - task: NodeTool@0 inputs: versionSpec: $(nodeVersion) displayName: 'Install Node.js' - script: | curl -fsSL https://get.pulumi.com | sh export PATH=$PATH:$HOME/.pulumi/bin npm ci displayName: 'Setup Pulumi and install dependencies' - script: | export PATH=$PATH:$HOME/.pulumi/bin pulumi up --yes --stack staging displayName: 'Deploy to staging' env: CHECKLY_ACCOUNT_ID: $(CHECKLY_ACCOUNT_ID) CHECKLY_API_KEY: $(CHECKLY_API_KEY) PULUMI_ACCESS_TOKEN: $(PULUMI_ACCESS_TOKEN) - stage: DeployProduction displayName: 'Deploy to Production' dependsOn: DeployStaging condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main')) jobs: - deployment: DeployProduction displayName: 'Deploy to Production' environment: production strategy: runOnce: deploy: steps: - task: NodeTool@0 inputs: versionSpec: $(nodeVersion) displayName: 'Install Node.js' - script: | curl -fsSL https://get.pulumi.com | sh export PATH=$PATH:$HOME/.pulumi/bin npm ci displayName: 'Setup Pulumi and install dependencies' - script: | export PATH=$PATH:$HOME/.pulumi/bin pulumi up --yes --stack prod displayName: 'Deploy to production' env: CHECKLY_ACCOUNT_ID: $(CHECKLY_ACCOUNT_ID) CHECKLY_API_KEY: $(CHECKLY_API_KEY) PULUMI_ACCESS_TOKEN: $(PULUMI_ACCESS_TOKEN) ``` ## CircleCI ### .circleci/config.yml ```yaml .circleci/config.yml theme={null} version: 2.1 orbs: node: circleci/node@5.1 jobs: test: docker: - image: cimg/node:18.17 steps: - checkout - node/install-packages: pkg-manager: npm - run: name: Run tests command: npm test - run: name: Run linting command: npm run lint preview: docker: - image: cimg/node:18.17 steps: - checkout - node/install-packages: pkg-manager: npm - run: name: Install Pulumi command: | curl -fsSL https://get.pulumi.com | sh echo 'export PATH=$PATH:$HOME/.pulumi/bin' >> $BASH_ENV - run: name: Preview changes command: | source $BASH_ENV pulumi preview --stack staging environment: CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID CHECKLY_API_KEY: $CHECKLY_API_KEY PULUMI_ACCESS_TOKEN: $PULUMI_ACCESS_TOKEN deploy-staging: docker: - image: cimg/node:18.17 steps: - checkout - node/install-packages: pkg-manager: npm - run: name: Install Pulumi command: | curl -fsSL https://get.pulumi.com | sh echo 'export PATH=$PATH:$HOME/.pulumi/bin' >> $BASH_ENV - run: name: Deploy to staging command: | source $BASH_ENV pulumi up --yes --stack staging environment: CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID CHECKLY_API_KEY: $CHECKLY_API_KEY PULUMI_ACCESS_TOKEN: $PULUMI_ACCESS_TOKEN deploy-production: docker: - image: cimg/node:18.17 steps: - checkout - node/install-packages: pkg-manager: npm - run: name: Install Pulumi command: | curl -fsSL https://get.pulumi.com | sh echo 'export PATH=$PATH:$HOME/.pulumi/bin' >> $BASH_ENV - run: name: Deploy to production command: | source $BASH_ENV pulumi up --yes --stack prod environment: CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID CHECKLY_API_KEY: $CHECKLY_API_KEY PULUMI_ACCESS_TOKEN: $PULUMI_ACCESS_TOKEN workflows: version: 2 test-and-deploy: jobs: - test - preview: requires: - test filters: branches: only: /.*/ tags: ignore: /.*/ - deploy-staging: requires: - test filters: branches: only: staging - deploy-production: requires: - test filters: branches: only: main ``` ## Security Best Practices ### Secret Management **Secure Storage** * Use CI/CD platform secret management * Never hardcode secrets in pipeline files * Rotate secrets regularly * Use least-privilege access **Role-Based Access** * Limit deployment permissions * Use separate accounts for different environments * Implement approval gates for production * Audit access regularly ### Security Checklist * [ ] Store secrets in CI/CD platform's secret management * [ ] Use environment-specific secrets * [ ] Rotate API keys regularly * [ ] Use least-privilege API keys * [ ] Never log or expose secrets * [ ] Implement approval gates for production * [ ] Use separate accounts for environments * [ ] Limit who can trigger deployments * [ ] Audit deployment permissions * [ ] Use branch protection rules * [ ] Validate code before deployment * [ ] Use trusted base images * [ ] Scan for vulnerabilities * [ ] Implement rollback procedures * [ ] Monitor deployment logs ## Troubleshooting ### Common Issues **Problem**: Pulumi or Checkly authentication fails **Solutions**: * Verify API keys are correctly set in secrets * Check Pulumi access token permissions * Ensure environment variables are properly configured * Test authentication locally first **Problem**: Pipeline lacks permissions to deploy **Solutions**: * Check CI/CD platform permissions * Verify Pulumi organization access * Ensure Checkly API key has required permissions * Review branch protection rules **Problem**: Resources already exist or conflict **Solutions**: * Use `pulumi refresh` to sync state * Check for manual changes in Checkly UI * Review resource naming conflicts * Use `pulumi import` for existing resources Always test your CI/CD pipeline in a development environment before deploying to production. Use feature branches and staging environments to validate changes. Never commit secrets or sensitive information to your repository. Always use your CI/CD platform's secret management features. # The Checkly Pulumi Provider Source: https://www.checklyhq.com/docs/integrations/iac/pulumi/overview Learn about the Checkly Pulumi Provider The [Checkly Pulumi provider](https://www.pulumi.com/registry/packages/checkly/) enables declaring your monitoring setup as code using [Pulumi](https://www.pulumi.com/). ## Getting started Learn how to create, configure and sync your checks by following this getting started guide. ### Pulumi CLI setup Before getting started and writing your monitoring setup in code, you have to set up Pulumi and its CLI. [Create a new Pulumi account](https://app.pulumi.com/signup) and [install the Pulumi CLI](https://www.pulumi.com/docs/get-started/install/). The `pulumi` command should then be available in your environment. ``` $ pulumi pulumi Pulumi - Modern Infrastructure as Code To begin working with Pulumi, run the `pulumi new` command: $ pulumi new This will prompt you to create a new project for your cloud and language of choice. ... ... ``` Create and copy a new personal access token in your Pulumi account at [app.pulumi.com/account/tokens](https://app.pulumi.com/account/tokens). Pulumi access token configuration screen showing a newly created token

Pulumi access token configuration screen showing a newly created token

Run `pulumi login` and provide the token. ``` $ pulumi login Manage your Pulumi stacks by logging in. Run `pulumi login --help` for alternative login options. Enter your access token from https://app.pulumi.com/account/tokens or hit to log in using your browser: ``` The CLI's `whoiam` command should return your username after a successful login. ``` $ pulumi whoami YOUR_USER_NAME ``` Now you're ready to create a new Pulumi project and start setting up your Checkly infrastructure. ### Create a new Pulumi project Create a new directory for your Pulumi project and navigate into it. ``` $ mkdir new-pulumi-checkly-project && cd $_ ``` Initialize a new Pulumi project with `pulumi new` and choose the minimal JavaScript template (`javascript`). ``` $ pulumi new javascript This command will walk you through creating a new Pulumi project. Enter a value or leave blank to accept the (default), and press . Press ^C at any time to quit. project name: (new-pulumi-checkly-project) project description: (A minimal JavaScript Pulumi program) Created project 'new-pulumi-checkly-project' Please enter your desired stack name. To create a stack in an organization, use the format / (e.g. `acmecorp/dev`). stack name: (dev) Created stack 'dev' Installing dependencies... added 96 packages, and audited 97 packages in 4s Finished installing dependencies Your new project is ready to go! ✨ To perform an initial deployment, run 'pulumi up' ``` Inspect the created files by running `ls` and make sure that a `Pulumi.yaml`, `package.json` and `index.js` file are available. ``` $ ls Pulumi.yaml node_modules package.json index.js package-lock.json ``` ### Define your Checkly account ID and API key To interact with [the underlying Checkly API](/docs/api-reference/overview), provide your Checkly account ID and API key by defining those in your environment. Your account ID is accessible at [app.checklyhq.com/settings/account/general](https://app.checklyhq.com/settings/account/general). Checkly account settings showing the account ID

Checkly account settings showing the account ID

``` export CHECKLY_ACCOUNT_ID=xxx ``` Create a new API key at [app.checklyhq.com/settings/user/api-keys](https://app.checklyhq.com/settings/user/api-keys). Checkly api key section showing a newly created API key

Checkly api key section showing a newly created API key

``` export CHECKLY_API_KEY=cu_xxx ``` Test that both values are available in your environment. ``` $ echo $CHECKLY_ACCOUNT_ID xxx $ echo $CHECKLY_API_KEY cu_xxx ``` > If you prefer storing your configuration alongside your Pulumi stack for easy multi-user access, use `pulumi config set`. ``` $ pulumi config set checkly:apiKey cu_xxx --secret $ pulumi config set checkly:accountId xxx ``` ### Install the Checkly Pulumi Provider Install the Checkly Pulumi provider (`@checkly/pulumi`) with `yarn` or `npm` to integrate it in your new project. ``` $ npm install @checkly/pulumi $ yarn add @checkly/pulumi ``` Open `index.js` and require it on top of the file. ```js theme={null} const checkly = require('@checkly/pulumi') ``` The exported `checkly` object provides resources such as `AlertChannel`, `Check`, and `CheckGroup`, enabling you to define and configure your entire Checkly monitoring setup in code. Add the following Check definition examples to the `index.js`. ### Examples #### How to create your first API check ```js theme={null} new checkly.Check('my-api-check', { name: 'Public SpaceX API', activated: true, frequency: 10, type: 'API', locations: ['eu-west-1', 'us-west-2'], tags: ['pulumi'], degradedResponseTime: 5000, maxResponseTime: 15000, request: { method: 'GET', url: 'https://api.spacexdata.com/v3', assertions: [ { source: 'STATUS_CODE', comparison: 'EQUALS', target: '200', }, { source: 'JSON_BODY', property: '$.project_name', comparison: 'EQUALS', target: 'SpaceX-API', }, ], }, useGlobalAlertSettings: true, }) ``` #### How to create your first Browser check ```javascript browser-check.js theme={null} new checkly.Check('my-browser-check-pulumi', { name: 'Google.com Playwright check', activated: true, frequency: 10, type: 'BROWSER', script: ` const { expect, test } = require('@playwright/test') test('visit page and take screenshot', async ({ page }) => { const response = await page.goto('https://checklyhq.com') expect(response.status()).toBeLessThan(400) await page.screenshot({ path: 'screenshot.jpg' }) })`, locations: ['eu-west-1', 'us-west-2'], tags: ['pulumi'], }) ``` ### Creating checks and applying changes Your `index.js` file should now include instructions to create one API and one Browser check. Initialize a deployment and apply your changes by running `pulumi up`. ``` $ pulumi up Previewing update (dev) View Live: https://app.pulumi.com/stefanjudis/new-pulumi-checkly-project/dev/previews/b30010eb-53a6-438a-864b-af29c2f1321f Type Name Plan pulumi:pulumi:Stack new-pulumi-checkly-project-dev + ├─ checkly:index:Check my-browser-check-pulumi create + └─ checkly:index:Check my-api-check-pulumi create Resources: + 2 to create 1 unchanged Do you want to perform this update? yes Updating (dev) View Live: https://app.pulumi.com/stefanjudis/new-pulumi-checkly-project/dev/updates/2 Type Name Status pulumi:pulumi:Stack new-pulumi-checkly-project-dev + ├─ checkly:index:Check my-api-check-pulumi created + └─ checkly:index:Check my-browser-check-pulumi created Resources: + 2 created Duration: 2s ``` Congratulations! Head to the Checkly dashboard to see the newly created checks. Checkly dashboard showing created checks

Checkly dashboard showing created checks

Checkly resources should be managed *either* through Pulumi *or* the Checkly UI, not both. Modifying Pulumi-managed resources via the UI, and vice-versa, is likely to cause issues. ## Additional resources Find more documentation on how to create groups, alerts and snippets online. * [The official Pulumi Provider documentation](https://www.pulumi.com/registry/packages/checkly/) * [The Pulumi Provider on GitHub](https://github.com/checkly/pulumi-checkly) * [Detailed examples in JavaScript and TypeScript](https://github.com/checkly/pulumi-checkly#examples) # Pulumi Setup Guide Source: https://www.checklyhq.com/docs/integrations/iac/pulumi/setup Detailed setup instructions for Pulumi CLI and Checkly integration This guide walks you through the complete setup process for using Pulumi with Checkly, from initial installation to your first deployment. ## Prerequisites Before starting, ensure you have: * **Node.js** (version 14 or higher) * **Git** for version control * **Checkly account** with API access * **Pulumi account** (free tier available) ## Step 1: Install Pulumi CLI Select your preferred installation method based on your operating system and package manager Run the appropriate installation command for your system Confirm Pulumi CLI is working correctly ### Installation Options ```bash install-pulumi-npm theme={null} npm install -g @pulumi/pulumi ``` ```bash install-pulumi-yarn theme={null} yarn global add @pulumi/pulumi ``` ```bash install-pulumi-brew theme={null} brew install pulumi/tap/pulumi ``` ```bash install-pulumi-curl theme={null} curl -fsSL https://get.pulumi.com | sh ``` ### Verify Installation ```bash verify-installation theme={null} pulumi version ``` You should see output similar to: ``` v3.100.0 ``` ## Step 2: Pulumi Account Setup ### Create Account 1. Visit [app.pulumi.com/signup](https://app.pulumi.com/signup) 2. Sign up with your preferred method (GitHub, GitLab, etc.) 3. Complete the account verification process ### Generate Access Token 1. Navigate to [app.pulumi.com/account/tokens](https://app.pulumi.com/account/tokens) 2. Click "Create token" 3. Give your token a descriptive name (e.g., "Checkly Integration") 4. Copy the generated token (you won't be able to see it again) Store your access token securely. You can use environment variables or a password manager to keep it safe. ### Login to Pulumi ```bash login-pulumi theme={null} pulumi login ``` When prompted, paste your access token or press Enter to use browser-based authentication. Verify your login: ```bash verify-login theme={null} pulumi whoami ``` ## Step 3: Checkly API Configuration ### Get Your Account ID 1. Log into [app.checklyhq.com](https://app.checklyhq.com) 2. Navigate to **Settings** → **Account** → **General** 3. Copy your **Account ID** ### Create API Key 1. Go to **Settings** → **User** → **API Keys** 2. Click "Create API Key" 3. Give it a descriptive name (e.g., "Pulumi Integration") 4. Copy the generated API key (starts with `cu_`) API keys have full access to your Checkly account. Keep them secure and never commit them to version control. ## Step 4: Project Setup ### Create Project Directory ```bash create-project theme={null} mkdir my-checkly-monitoring && cd my-checkly-monitoring ``` ### Initialize Pulumi Project ```bash init-pulumi-project theme={null} pulumi new javascript ``` Follow the prompts: * **Project name**: `my-checkly-monitoring` (or your preferred name) * **Project description**: `Checkly monitoring infrastructure as code` * **Stack name**: `dev` (or your preferred environment name) ### Install Checkly Provider ```bash install-checkly-npm theme={null} npm install @checkly/pulumi ``` ```bash install-checkly-yarn theme={null} yarn add @checkly/pulumi ``` ## Step 5: Configure Authentication ### Option 1: Environment Variables (Recommended for Development) Add to your shell profile (`.bashrc`, `.zshrc`, etc.): ```bash setup-env-vars theme={null} export CHECKLY_ACCOUNT_ID=your_account_id export CHECKLY_API_KEY=your_api_key ``` Reload your shell or source the profile: ```bash reload-shell theme={null} source ~/.bashrc # or ~/.zshrc ``` ### Option 2: Pulumi Configuration (Recommended for Teams) ```bash setup-pulumi-config theme={null} pulumi config set checkly:apiKey your_api_key --secret pulumi config set checkly:accountId your_account_id ``` Using Pulumi configuration is better for team environments as it stores settings with your stack and can be shared across team members. ### Verify Configuration Test that your credentials are accessible: ```bash verify-env-vars theme={null} echo $CHECKLY_ACCOUNT_ID echo $CHECKLY_API_KEY ``` Or check Pulumi config: ```bash check-pulumi-config theme={null} pulumi config ``` ## Step 6: Create Your First Check ### Update index.js Replace the contents of `index.js` with: ```javascript index.js theme={null} const checkly = require('@checkly/pulumi') // Create a simple API check new checkly.Check('hello-world-api', { name: 'Hello World API', activated: true, frequency: 10, type: 'API', locations: ['eu-west-1'], tags: ['pulumi', 'example'], request: { method: 'GET', url: 'https://httpbin.org/status/200', assertions: [ { source: 'STATUS_CODE', comparison: 'EQUALS', target: '200', }, ], }, useGlobalAlertSettings: true, }) ``` ## Step 7: Deploy Your Infrastructure ### Preview Changes ```bash preview-changes theme={null} pulumi preview ``` This shows you what resources will be created without making changes. ### Deploy ```bash deploy-infrastructure theme={null} pulumi up ``` When prompted, type `yes` to confirm the deployment. Congratulations! You've successfully set up Pulumi with Checkly and deployed your first monitoring check. ## Step 8: Verify Deployment 1. Visit your [Checkly dashboard](https://app.checklyhq.com) 2. Navigate to **Checks** 3. You should see your new "Hello World API" check 4. The check should be running and showing as "Passing" ## Next Steps Explore more complex examples and configurations Learn best practices for managing monitoring as code Set up automated deployments in your CI/CD pipeline Common issues and solutions ## Troubleshooting ### Common Issues If you see authentication errors, verify: * Your Checkly API key is correct and active * Your account ID is correct * Environment variables are properly set * You're logged into Pulumi (`pulumi whoami`) If the Checkly provider isn't found: * Ensure you've installed `@checkly/pulumi` * Check your `package.json` for the dependency * Run `npm install` or `yarn install` If you get permission errors: * Verify your API key has the necessary permissions * Check that your account ID is correct * Ensure you're using the right Checkly account # Alert channels and settings for Terraform Source: https://www.checklyhq.com/docs/integrations/iac/terraform/alerting Configure alert settings and alert channels for your Terraform-managed monitoring setup Alerting is organised around alert settings and alert channels. ## Alert settings [Alert settings](/docs/communicate/alerts/overview) determine *when* and how many times alerts should be sent out. You can set the alert settings at check level: ```terraform theme={null} resource "checkly_check" "get-books" { name = "GET /books" type = "API" // ... alert_settings { escalation_type = "RUN_BASED" // Whether to alert after a certain number of runs or time run_based_escalation { failed_run_threshold = 1 // (RUN_BASED escalation only) after how many failed runs to send an alert } time_based_escalation { minutes_failing_threshold = 5 // (TIME_BASED escalation only) after how many minutes spent in failing state to send an alert } ssl_certificates { enabled = true // Whether the SSL certificate will be checked for expiry alert_threshold = 30 // At which number of days remaining before expiry should the alert be sent } reminders { amount = 2 // How many reminders to send after the first alert interval = 5 // How many minutes to wait between reminders } parallel_run_failure_threshold { enabled = true // Applicable only for checks scheduled in parallel in multiple locations percentage = 50 // What percentage of regions needs to fail to trigger a failure alert, supported values: 10, 20, 30, 40, 50, 60, 70, 80, 90 & 100 } } // ... } ``` Or at group level: ```terraform theme={null} resource "checkly_check_group" "key-shop-flows" { name = "Key Shop Flows" activated = true // ... alert_settings { escalation_type = "RUN_BASED" run_based_escalation { failed_run_threshold = 1 } time_based_escalation { minutes_failing_threshold = 5 } ssl_certificates { enabled = true alert_threshold = 30 } reminders { amount = 2 interval = 5 } parallel_run_failure_threshold { enabled = true percentage = 50 } } } ``` ## Alert channels [Alert channels](/docs/communicate/alerts/overview) determine *how* (as in "to which alert systems") alerts are sent out. They can be created as standalone resources: ```terraform theme={null} resource "checkly_alert_channel" "email_ac" { email { address = "john@example.com" // The email address to notify } send_recovery = true // Whether email will be sent when a check recovers send_failure = false // Whether email will be sent when a check fails send_degraded = true // Whether email will be sent when a check's performance degrades } ``` Checkly supports a variety of alert channels, from email and SMS to Pagerduty and custom webhooks. For all available alert channels, see the corresponding [resource page](https://registry.terraform.io/providers/checkly/checkly/latest/docs/resources/alert_channel). ### Alert channel subscriptions Checks or groups are subscribed to one or more alert channels: ```terraform theme={null} resource "checkly_check" "get-books" { name = "GET /books" type = "API" // ... alert_channel_subscription { // This subscribes the check to the alert channel channel_id = checkly_alert_channel.email_ac.id activated = true } // ... } resource "checkly_check_group" "key-shop-flows" { name = "Key Shop Flows" activated = true // ... alert_channel_subscription { // This subscribes the group to the alert channel channel_id = checkly_alert_channel.email_ac.id activated = true } } ``` Checks and groups need to be *explicitly* subscribed to an alert channel as shown above - they will *not* be autosubscribed. Checks which are part of a group inherit the group's alert subscriptions. If you know a check will be part of a group, you can avoid needlessly subscribing it to alert channels and just handle everything at the group level. # CI/CD pipelines for Terraform Source: https://www.checklyhq.com/docs/integrations/iac/terraform/ci-cd Integrate Terraform with CI/CD tools for automated monitoring setup deployment If you are using or planning to use Terraform to manage your monitoring resources, chances are you will be using it in some sort of shared pipeline. This page aims to show how to get up and running with different CI/CD tools. > The Terraform and the Checkly Terraform provider work seamlessly with all major CI servers. Even if your tool of choice is not listed below, Terraform and the provider will very likely still integrate with it without issues. ## Handling remote state Terraform relies on its stored state to know which resources it is supposed to manage and what to do to apply your configuration. If you are applying Terraform configurations from different machines, or are working on your setup with your colleagues, you need to make sure the Terraform state stays in sync. Failure to do that will lead to duplicated resources and other critical issues with your checks and other managed resources. Terraform enables [remote state](https://www.terraform.io/language/state/remote) management using [backends](https://www.terraform.io/language/settings/backends). If you are new to Terraform and plan to use it with Checkly together with your team or the rest of your organization, make sure you set up a remote backend to enable frictionless collaboration. ## Terraform Cloud [Terraform Cloud](https://terraform.io/cloud) allows you to automate setting up your monitoring resources on Checkly, while also [taking care](https://www.terraform.io/cloud-docs/workspaces/state) of remote state management for you, with no configuration needed. checkly terraform cloud integration diagram

checkly terraform cloud integration diagram

Getting started with Terraform Cloud is simple: you only need to create a new workspace and [link it to version control](https://www.terraform.io/cloud-docs/vcs). terraform cloud new vcs project

Our linked repository will contain our Terraform code, for example: ```terraform theme={null} variable "checkly_api_key" {} variable "checkly_account_id" {} terraform { required_providers { checkly = { source = "checkly/checkly" version = "~> 1.0" } } } provider "checkly" { api_key = var.checkly_api_key account_id = var.checkly_account_id } // Your resources here ``` While you can apply your Terraform configuration on-demand by clicking `Actions` and `Start new plan`, you will likely want to have your workspace set up to automatically apply it on every push to the default branch of the linked repository. You can enable that through the `Auto apply` option under the `General` settings panel of your Terraform Cloud workspace. terraform cloud auto apply

Any new push to the default branch of the repository will now result in your latest Terraform configuration being applied, which in turn means your Checkly monitoring setup will be updated. terraform cloud apply report

> This is a v1 page. We will be adding further guides on how to integrate with other CI servers going forward. # Command line triggers for Terraform Source: https://www.checklyhq.com/docs/integrations/iac/terraform/command-line-triggers Set up command line triggers to run checks from CI/CD pipelines or programmatically [Command line triggers](/docs/cli/checkly-trigger) enable you to call a check from a CI/CD pipeline, a bash shell or programmatically in your code. You can specify command line triggers as standalone resources at check or group level: ```terraform theme={null} resource "checkly_trigger_check" "test-trigger-check" { check_id = "c1ff95c5-d7f6-4a90-9ce2-1e605f117592" // The id of the check to be triggered } output "test-trigger-check-url" { value = checkly_trigger_check.test-trigger-check.url // This will output the trigger url } resource "checkly_trigger_group" "test-trigger-group" { // The id of the group to be triggered group_id = "215" } output "test-trigger-group-url" { value = checkly_trigger_group.test-trigger-group.url // This will output the trigger url } ``` You can see all the configuration options for [group triggers](https://registry.terraform.io/providers/checkly/checkly/latest/docs/resources/trigger_group), as well as more examples [check triggers](https://registry.terraform.io/providers/checkly/checkly/latest/docs/resources/trigger_check), on the official Terraform registry documentation page. # Dashboards in Terraform Source: https://www.checklyhq.com/docs/integrations/iac/terraform/dashboards Create and configure public dashboards to display check metrics and status using Terraform [Dashboards](/docs/communicate/dashboards/overview) allow you to display checks and their related metrics on a single page. They come with their own dedicated resource: ```terraform theme={null} resource "checkly_dashboard" "dashboard-main" { custom_url = "danube" // A unique subdomain name under "checklyhq.com" custom_domain = "status.danube.com" // A custom user domain logo = "https://www.danube-web.shop/logo.png" // URL pointing to an image/logo for the page header = "My dashboard" // What text to display at the top of your dashboard refresh_rate = 60 // How often to refresh the dashboard in seconds paginate = false // Determines if pagination is on or off pagination_rate = 30 // How often to trigger pagination in seconds hide_tags = false // Whether to show or hide the tags on the dashboard width = "FULL" // Determines whether to use the full screen or focus in the center isPrivate = "false" // Sets the dashboard to private. Will return a `key` to be used as password tags = [ // One or more tags that filter which checks to display on the dashboard "auto" ] } ``` > By default, if no check is specified, all checks in the account will be shown in the dashboard. If instead you want to show specific checks, you will need to [use tags](/docs/communicate/dashboards/configuration). You can see all the configuration options for checks, as well as more examples, on the official Terraform registry [documentation page](https://registry.terraform.io/providers/checkly/checkly/latest/docs/resources/dashboard). ### Private Dashboards > Available as of `v1.6.6` You can control your dashboards private/public status via Terraform by setting the `isPrivate` boolean to `true`. If activated, upon applying your terraform configuration changes, you will be returned a `read-only` `key` value for the dashboard which represents the password you can use to login via the browser. The password is stored in your `tfstate` file. # Service maintenance windows in Terraform Source: https://www.checklyhq.com/docs/integrations/iac/terraform/maintenance-windows Schedule planned maintenance windows to prevent checks from running during specific times With [maintenance windows](/docs/communicate/maintenance-windows/overview) you can schedule planned maintenance for whatever you are monitoring in order to prevent your checks from running at specific times. Maintenance windows have their own resource: ```terraform theme={null} resource "checkly_maintenance_windows" "maintenance-monthly" { name = "Monthly maintenance" // The name of the maintenance window starts_at = "2022-08-24T00:00:00.000Z" // The start date of the maintenance window ends_at = "2022-08-25T00:00:00.000Z" // The end date of the maintenance window repeat_unit = "MONTH" // The repeat strategy for the maintenance window tags = [ // The tags of the checks and groups maintenance window should apply to "auto" ] } ``` You can see all the configuration options for maintenance windows, as well as more examples, on the official Terraform registry [documentation page](https://registry.terraform.io/providers/checkly/checkly/latest/docs/resources/maintenance_windows). # Getting started with the Terraform provider for Checkly Source: https://www.checklyhq.com/docs/integrations/iac/terraform/overview Learn how to use the Checkly Terraform provider to declare your monitoring setup as code using HashiCorp Terraform The [Checkly Terraform provider](https://github.com/checkly/terraform-provider-checkly) enables you to declare your monitoring setup as code using [HashiCorp Terraform](https://www.terraform.io/). You can get started with it in a matter of minutes by following the steps shown below. If you prefer, you can also clone our [sample repository](https://github.com/checkly/checkly-terraform-getting-started) and play around with the resources on your own. For in-depth information on Terraform, please see HashiCorp's [official documentation](https://developer.hashicorp.com/terraform/docs). > Looking for Monitoring as Code but don't want to use Terraform? Give our [CLI solution](/docs/cli/overview) a try. ## Installation To get started, you first need to install [Terraform](https://www.terraform.io/downloads). Next, switch to a new folder for your project, and create your `main.tf` file: ```bash Terminal theme={null} mkdir terraform-checkly-getting-started && cd $_ && touch main.tf ``` Start by adding the following to your `main.tf`: ```terraform main.tf theme={null} terraform { required_providers { checkly = { source = "checkly/checkly" version = "~> 1.0" } } } variable "checkly_api_key" {} variable "checkly_account_id" {} provider "checkly" { api_key = var.checkly_api_key account_id = var.checkly_account_id } ``` This is all the code you need to set up the Checkly Terraform provider. You now need to set your credentials to give the provider access to your Checkly account. First, you will need an API Key for your Checkly user. Go to the [API keys tab](https://app.checklyhq.com/settings/user/api-keys) in your user settings and click `Create API key`. api key tab in checkly settings

Get your User API key and add it to your env using your terminal: ```bash Terminal theme={null} export TF_VAR_checkly_api_key=cu_xxx ``` You also need to set your target account ID, which you can find under your [account settings](https://app.checklyhq.com/settings/account/general). account id in checkly settings

If you don't have access to account settings, please contact your account owner/admin. ```bash Terminal theme={null} export TF_VAR_checkly_account_id=xxx ``` Running `terraform init` will install the Checkly Terraform provider for you, as well as initialising your project. The output will look similar to the following: ```bash Terminal theme={null} $ terraform init Initializing the backend... Initializing provider plugins... - Finding checkly/checkly versions matching "~> 1.0"... - Installing checkly/checkly v1.4.3... - Installed checkly/checkly v1.4.3 (signed by a HashiCorp partner, key ID 4E5AC4D95E185A57) ... Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` You can now start adding resources to your file. You can check the official documentation to see [available parameters](https://registry.terraform.io/providers/checkly/checkly/latest/docs/resources/check) for each resource type. ## Your first browser check As an example, you could add a basic browser check resource at the bottom of your `main.tf`. ```terraform main.tf theme={null} resource "checkly_check" "browser-check-1" { name = "Example check" type = "BROWSER" activated = true frequency = 5 locations = [ "eu-central-1", "us-west-1" ] retry_strategy = { type = "LINEAR" } script = < { await page.goto('https://playwright.dev/'); // Expect a title "to contain" a substring. await expect(page).toHaveTitle(/Playwright/); }); EOT } ``` ## Applying changes You can now have Terraform apply your changes on Checkly by running `terraform apply`. Terraform will draft a plan and ask you to confirm by typing `yes`. Once that is done, Terraform will go ahead and create the resources you have specified (one browser check, so far) for you. ```bash Terminal theme={null} $ terraform apply Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # checkly_check.browser-check-1 will be created + resource "checkly_check" "browser-check-1" { + activated = true + degraded_response_time = 15000 + retry_strategy { + base_backoff_seconds = 60 + max_duration_seconds = 600 ... } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes checkly_check.browser-check-1: Creating... checkly_check.browser-check-1: Creation complete after 1s [id=cc143901-fbae-4942-91fe-4b99b0250567] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. ``` Your Checkly account should now show the first check up and running: checkly dashboard with running check

## Adding an API check You can also add an API check to monitor an API endpoint: ```terraform main.tf theme={null} resource "checkly_check" "example-check" { name = "Example check" type = "API" activated = true frequency = 1 locations = [ "us-west-1", "eu-central-1" ] request { url = "https://api.example.com/" follow_redirects = true skip_ssl = false assertion { source = "STATUS_CODE" comparison = "EQUALS" target = "200" } } } ``` You can now choose to re-run `terraform apply` or wait until we have added an alert channel and have Terraform create both resources at once. ## Setting up alerts Now that you have your checks set up, you'll want to set up an alert channel for Checkly to promptly inform you when one of them fails. For example, you could create an email channel: ```terraform main.tf theme={null} resource "checkly_alert_channel" "email_ac" { email { address = "jane.doe@email.com" // Your email address here } send_recovery = true send_failure = true send_degraded = false } ``` Next, subscribe each check to the alert channel: ```terraform main.tf theme={null} resource "checkly_check" "browser-check-1" { name = "Example check" type = "BROWSER" // ... alert_channel_subscription { channel_id = checkly_alert_channel.email_ac.id activated = true } // ... } resource "checkly_check" "example-check" { name = "Example check" type = "API" // ... alert_channel_subscription { channel_id = checkly_alert_channel.email_ac.id activated = true } // ... } ``` You can now run `terraform apply` for the last time and have Terraform create your checks and alert channel for you. checkly dashboard with running check

Congratulations! You have created your first Checkly monitoring setup using Terraform. Checkly resources should be managed *either* through Terraform *or* through the Checkly UI, not both. Modifying Terraform-managed resources via the UI, and vice-versa, is likely to cause issues. # Environment Variables in Terraform Source: https://www.checklyhq.com/docs/integrations/iac/terraform/snippets-variables Manage reusable code snippets and environment variables to reduce duplication in your monitoring setup Checkly provides resources to eliminate code duplication and reduce maintenance overhead in your monitoring setup. ## Code snippets [Code snippets](/docs/detect/synthetic-monitoring/browser-checks/snippets) allow you to reuse code across multiple checks without duplication. They work with [setup & teardown scripts](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown#reusable-code-snippets) in API checks and [browser checks](/docs/detect/synthetic-monitoring/browser-checks/snippets). ```terraform theme={null} resource "checkly_snippet" "login-procedure" { name = "Login" script = < ```terraform theme={null} resource "checkly_snippet" "login-procedure" { name = "Login" script = file("${path.module}/snippets/login.js") } ``` Store your snippet in a separate file for better organization and syntax highlighting. ## Environment variables [Environment variables](/docs/platform/variables) store sensitive data like credentials or configuration values that can be used across checks and groups. ```terraform theme={null} resource "checkly_check" "user-login" { name = "User Login Flow" type = "BROWSER" activated = true frequency = 5 environment_variables = { TEST_EMAIL = "user@example.com" TEST_PASSWORD = "secure-password" API_BASE_URL = "https://api.example.com" } locations = ["us-west-1", "eu-central-1"] script = file("${path.module}/scripts/login.spec.js") } ``` Variables defined at check level are only available to that specific check. ```terraform theme={null} resource "checkly_check_group" "ecommerce-flows" { name = "E-commerce Flows" activated = true environment_variables = { TEST_EMAIL = "user@example.com" TEST_PASSWORD = "secure-password" API_BASE_URL = "https://api.example.com" } locations = ["us-west-1", "eu-central-1"] concurrency = 3 } ``` Variables defined at group level are inherited by all checks within that group. Account-level environment variables via Terraform are coming soon. For now, use check or group-level variables for your monitoring setup. # Testing Terraform scripts locally Source: https://www.checklyhq.com/docs/integrations/iac/terraform/testing-scripts-locally Run Playwright Test scripts locally for faster development and debugging of browser checks Having the possibility to run Playwright Test scripts for your browser checks locally will allow you to develop them faster and with more confidence. ## Enabling local script execution You can easily enable local execution by storing your browser check scripts in local folders and passing the path to the right script: ```terraform theme={null} resource "checkly_check" "e2e-checkout" { name = "Checkout Flow" type = "BROWSER" activated = true should_fail = false frequency = 1 use_global_alert_settings = true locations = [ "us-west-1", "eu-central-1" ] retry_strategy = { type = "LINEAR" } script = file("${path.module}/scripts/checkout.spec.ts") // Or .js - our script is contained in this file } ``` Basic checks written with `@playwright/test` will run locally with `npx playwright test` and remotely on Checkly without any modifications: ```ts basic.spec.ts theme={null} import { test, expect } from '@playwright/test' test('Should load the web store', async ({ page }) => { await page.goto('https://danube-web.shop') await expect(page).toHaveTitle('danube-store') }) ``` ```js basic.spec.js theme={null} const { test, expect } = require('@playwright/test') test('Should load the web store', async ({ page }) => { await page.goto('https://danube-web.shop') await expect(page).toHaveTitle('danube-store') }) ``` If your script is using [Page Object Models](https://playwright.dev/docs/pom) or imports other files, you can take advantage of Checkly's [code snippets](/docs/integrations/iac/terraform/snippets-variables). Consider the following directory structure: ``` scripts / |--- snippets / | |--- shoppingCart.ts // or .js |--- checkout.spec.ts // or .js ``` `scripts/snippets/shoppingCart.{js,ts}` contains a Page Object Model class encapsulating the logic for the store's shopping cart page: ```ts shopping.spec.ts theme={null} import { type Locator, type Page } from '@playwright/test' export class ShoppingCart { shoppingCartButton: Locator summary: Locator constructor ({ page }: { page: Page }) { this.shoppingCartButton = page.locator('#cart') this.summary = page.locator('.cart') } async clickShoppingCartButton () { return this.shoppingCartButton.click() } } ``` ```js shopping.spec.js theme={null} export class ShoppingCart { constructor ({ page }) { this.shoppingCartButton = page.locator('#cart') this.summary = page.locator('.cart') } async clickShoppingCartButton () { return this.shoppingCartButton.click() } } ``` `scripts/checkout.spec.{js,ts}` file is your Checkly check with the following content: ```ts checkout.spec.ts theme={null} import { test, expect } from '@playwright/test' import { ShoppingCart } from './snippets/shoppingCart' test('Shopping cart should be empty by default', async ({ page }) => { await page.goto('https://danube-web.shop') const shoppingCart = new ShoppingCart({ page }) await shoppingCart.clickShoppingCartButton() await expect(shoppingCart.summary).toContainText('Your shopping cart is empty') }) ``` ```js checkout.spec.js theme={null} const { test, expect } = require('@playwright/test') const { ShoppingCart } = require('./snippets/shoppingCart') test('Shopping cart should be empty by default', async ({ page }) => { await page.goto('https://danube-web.shop') const shoppingCart = new ShoppingCart({ page }) await shoppingCart.clickShoppingCartButton() await expect(shoppingCart.summary).toContainText('Your shopping cart is empty') }) ``` As you can see, it imports the `ShoppingCart` class from the `./snippets/shoppingCart` directory and uses its methods and locators within the test. This setup will work out of the box locally with `npx playwright test`. If you'd like to use the external files in Checkly, you'll need to declare them as a `checkly_snippet` resource in your `.tf` file: ```terraform theme={null} resource "checkly_snippet" "shopping_cart" { name = "shoppingCart" // This will be the name of your file in Checkly! script = file("${path.module}/scripts/snippets/shoppingCart.ts") // Or .js - Your script is contained in this file } ``` > What's worth noting > > * Checkly will infer the name for the snippet file based on the `name` property of the resource schema. Hence, it's best to name it the same as the local file. > * The `./snippets/` path is where Checkly will make your snippets available. Use this directory in your Terraform setup to keep things consistent. If you'd like to store the snippet files in a different directory, using environment variables to define the paths for local and remote execution could be a solution. # Common issues with the Terraform provider Source: https://www.checklyhq.com/docs/integrations/iac/terraform/troubleshooting Troubleshoot common issues when using the Checkly Terraform provider This page covers common issues users have run into while using the Checkly Terraform provider, together with their likely root cause. ## Error: 402 payment required You might have reached the maximum number of checks you can create on your account. Contact Checkly Support via our chat widget to talk about options. ## Alert channel has 0 out of 0 subscribers You might have forgotten to [subscribe checks and groups](/docs/integrations/iac/terraform/alerting#alert-channel-subscriptions) to your alert channel. ## Check seems to run from the wrong location(s) Different things might be happening: * If the check is part of a group, group-level settings will decide which location set is actually used to run all the checks in that group. Check again whether your check is part of a group by checking if its resource contains the `group_id` parameter. * If you don’t select more than one location and you’ve disabled retrying checks from the same location, we will pick a random location when retrying checks. ## Alerts are not sent for one or more checks / the wrong channels are used The checks you are managing could be part of a group. If so, the group-level alert channel subscriptions will override check-level ones. This is easily fixed by declaring the alert channel subscriptions at group-level only. # Send Alerts to FireHydrant Source: https://www.checklyhq.com/docs/integrations/incident-management/firehydrant Integrate with FireHydrant to create and resolve incidents from Checkly alerts Checkly integrates with [FireHydrant](https://firehydrant.io/) to create and resolve incidents in your FireHydrant installation. More specifically, Checkly will: * Alert you on Slack when a check fails, allowing you to create an incident on FireHydrant with one click. * Close incidents automatically when a failing check recovers. Assuming you already have [set up Slack on FireHydrant](https://docs.firehydrant.com/), the integration can be configured as follows: 1. Log in to FireHydrant and navigate to `Integrations > Connected & available`, then select Checkly. firehydrant integration step 1

2. Next, copy the generated URL to your clipboard. firehydrant integration step 2

3. Log in to Checkly and navigate to `Alert Settings`. Click the `Add more channels` button, find FireHydrant on the list, and click `Add channel`. firehydrant integration step 3

4. Enter a name of your choosing for the alert channel, together with the URL you copied from FireHydrant. Make sure that the right checks are subscribing to the channel and that the `Send when` rules are correctly set, then hit `Save FireHydrant Webhook`. firehydrant integration step 4

You are all set! Now, when a check fails, you will see an alert in your configured Slack channel: firehydrant integration step 5

At this point, you can choose to either create an incident on FireHydrant or dismiss the alert. In case the related check recovers, the incident will be closed. firehydrant integration step 6

Congratulations, you have successfully integrated Checkly with FireHydrant! # Send Alerts to ilert Source: https://www.checklyhq.com/docs/integrations/incident-management/ilert Configure ilert integration to monitor incidents from Checkly alerts Checkly integrates with [ilert](https://www.ilert.com/) to monitor your incidents in your ilert installation. Let's get started! 1. Log in to ilert and navigate to `Alert sources > Create a new alert source`. ilert integration step 1

2. Next, create a new alert source with Checkly as integration type. ilert integration step 2

3. Copy the generated API key to your clipboard. ilert integration step 3

4. Log in to Checkly and navigate to `Alert Settings`. Click the `Add more channels` button, find ilert icon on the list, and click `Add channel`. ilert integration step 4

5. Enter a name of your choosing for the alert channel, together with the ilert URL and your API key (`https://api.ilert.com/api/v1/events/checkly/{api-key}`). Make sure that the right checks are subscribing to the channel and that the `Send when` rules are correctly set, then hit `Save ilert Webhook`. ilert integration step 5

Congratulations, you've successfully integrated Checkly with ilert! # Send Alerts to Incident.io Source: https://www.checklyhq.com/docs/integrations/incident-management/incidentio Configure Incident.io integration to create and resolve incidents from Checkly alerts Checkly integrates with [Incident.io](https://incident.io/) to monitor your alerts and create / resolve incidents in your Incident.io account. Let's get started! 1. Log in to Incident.io and navigate to **Alerts > Sources**, click **New Alert Source** and select "Checkly"." incident.io integration step 1

2. Take note of the URL and API key that Incident.io provides you with, and head over to [Checkly > Alerts](https://app.checklyhq.com/alerts/settings), click **Add more channels** and select **Incident.io**. incident.io integration step 2

3. Copy and paste in the URL and API key and hit **Test Incident.io alert** to verify the connection. On the Incident.io side, you should now see a test alert pop up! incident.io integration step 3

4. We recommend one last step. Create a new custom attribute on the Incident.io side to grab the Checkly result link from the JSON payload. This will help reference what check result triggered an alert. incident.io integration step 4

5. Now you are all set up! When an alert is triggered, an incident will be created in Incident.io with the Checkly check result link attached. incident.io integration step 5

# Send Alerts to Opsgenie Source: https://www.checklyhq.com/docs/integrations/incident-management/opsgenie Learn how to integrate Checkly with Opsgenie and deliver failure, degradation, and recovery events to any Opsgenie team Checkly integrates with [Opsgenie](https://opsgenie.com) and can deliver failure, degradation, and recovery events to any Opsgenie team. You can add as many Opsgenie channels as you wish. After setting up the integration, Checkly will: 1. Create an alert in Opsgenie when a check fails. 2. Send the most recent result details when a check degrades. 3. Resolve alerts when a check recovers. **To get started, take the following steps:** 1. First, you will need to create an **API integration** in Opsgenie. Log in to your Opsgenie account, and determine which region you are using. At the moment Opsgenie offers two [regions:](https://docs.opsgenie.com/docs/european-service-region) `US` and `EU`. > 💡 If your Opsgenie URL looks something like [https://app.eu.opsgenie.com](https://app.eu.opsgenie.com), > you are using the `EU` region. Go to the **Teams** tab and choose a team you want to integrate with Checkly, then navigate to your team's **Integrations** tab and click the "Add integration" button. setup checkly opsgenie integration step 1

setup checkly opsgenie integration step 1

2. Select the **API** integration from the list of integrations available in Opsgenie. > 💡 You can use the `REST API` query in the search box, to quickly find the **API** integration. setup checkly opsgenie integration step 2

setup checkly opsgenie integration step 2

3. Choose a name for your integration, copy the **API Key** and click "Save Integration" at the bottom of the screen to save your changes.\\ setup checkly opsgenie integration step 3

setup checkly opsgenie integration step 3

4. Log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find Opsgenie on the list, and click "Add channel" next to it to create a new Opsgenie channel. setup checkly opsgenie integration step 4

setup checkly opsgenie integration step 4

5. Choose a name for your Opsgenie channel, paste your **API Key** and select your Opsgenie region (`EU` for EU based customers, `US` for the rest of the world). > 💡 In OpsGenie you can configure other options like > [alert priority](https://docs.opsgenie.com/docs/alert-priority-settings). > 💡 On Checkly, you can specify which [check events](/docs/communicate/alerts/channels) will be sent over to Opsgenie. setup checkly opsgenie integration step 5

setup checkly opsgenie integration step 5

6. You can click the "Test API Key" button to test your integration, if everything is correct you should see a test alert created on your Opsgenie dashboard. **To finish the setup process**, back in Checkly click the "Save Opsgenie integration" button. setup checkly opsgenie integration step 6

setup checkly opsgenie integration step 6

Congratulations! You have successfully integrated Checkly with your Opsgenie team. # Send Alerts to PagerDuty Source: https://www.checklyhq.com/docs/integrations/incident-management/pagerduty Learn how Checkly integrates seamlessly with PagerDuty, delivering real-time failure and recovery alerts to your PagerDuty account **Using environment variables and secrets in this channel's config** Any literal-secret field in this alert channel — webhook URL, API key, bearer token, service key, etc. — accepts a [variable](/docs/platform/variables) or [secret](/docs/platform/secrets) reference using the familiar Handlebars syntax: `{{MY_VARIABLE}}`. You can also promote a value to a secret directly from the field: type the literal value, click the lock icon next to the input, and choose a name. The value is stored as an encrypted account secret and is never re-displayed in the UI. The PagerDuty channel's `serviceKey` (integration key) accepts environment-variable references — e.g. `{{PAGERDUTY_SERVICE_KEY}}` — so you can store the integration key as an account secret instead of inline in the channel config. Checkly integrates with [Pagerduty](https://pagerduty.com) and can deliver all failure and recovery events to your Pagerduty account. After setting up the integration, Checkly will: 1. Trigger alerts in Pagerduty when a check fails. 2. Resolve alerts when a check recovers. Setup is as simple as following the three-step Pagerduty connect process. You can add as many Pagerduty channels as you wish. 1. Navigate to the **alert settings** tab on the account screen and click the 'Add channels' button. setup checkly pagerduty integration step 1

setup checkly pagerduty integration step 1

2. Clicking the **Alert with Pagerduty** button will take you to a Pagerduty Connect screen. Provide your credentials and click **Sign In** to allow Checkly to connect with Pagerduty. set up checkly pagerduty integration step 2

set up checkly pagerduty integration step 2

3. On the next screen you can hook up Checkly to one or more existing services. If you select multiple services, we will create multiple dedicated channels so you have more flexibility of muting, editing and managing the channels. Click **Connect** to save your settings and redirect you back to Checkly. set up checkly pagerduty integration step 3

set up checkly pagerduty integration step 3

4. Back in Checkly, you should see your Pagerduty **integration credentials** reflected in the alert settings. Don't forget to hit **Save Pagerduty channel**. set up checkly pagerduty integration step 4

set up checkly pagerduty integration step 4

5. Checkly will trigger an incident in Pagerduty when checks fail and also mark them as resolved when the checks are passing again. set up checkly pagerduty integration step 5

set up checkly pagerduty integration step 5

## Advanced PagerDuty Configuration ### PagerDuty Alert Mapping Configure how Checkly alerts map to PagerDuty incidents: ```yaml theme={null} PagerDuty Configuration: Service: Production Monitoring Integration Type: Events API v2 Alert Mapping: ALERT_FAILURE → Critical Incident ALERT_DEGRADED → Warning Incident ALERT_RECOVERY → Resolve Incident Incident Details: Title: "{{ALERT_TITLE}}" Description: "Check {{CHECK_NAME}} from {{RUN_LOCATION}}" Severity: Based on check tags and alert type Component: Derived from check name or group Custom Fields: - Response Time: {{RESPONSE_TIME}}ms - Check ID: {{CHECK_ID}} - Location: {{RUN_LOCATION}} ``` ### Advanced PagerDuty Features Leverage PagerDuty's advanced incident management capabilities: ```json theme={null} { "incident": { "type": "incident", "title": "{{ALERT_TITLE}}", "service": { "id": "P123ABC", "type": "service_reference" }, "priority": { "id": "{{#contains TAGS 'critical'}}P1{{else}}P2{{/contains}}", "type": "priority_reference" }, "body": { "type": "incident_body", "details": "Check {{CHECK_NAME}} failed from {{RUN_LOCATION}}.\n\nResponse Time: {{RESPONSE_TIME}}ms\nError: {{CHECK_ERROR_MESSAGE}}\n\nView details: {{RESULT_LINK}}" }, "escalation_policy": { "id": "{{#contains CHECK_NAME 'payment'}}P456DEF{{else}}P789GHI{{/contains}}", "type": "escalation_policy_reference" } } } ``` ### PagerDuty Best Practices * **Service Organization**: Create separate PagerDuty services for different applications or teams * **Escalation Policies**: Set up appropriate escalation chains based on check criticality * **Priority Mapping**: Use check tags to automatically assign appropriate incident priorities * **Integration Keys**: Use different integration keys for different environments (production, staging) * **Custom Fields**: Include relevant context like response times, error messages, and check locations # Send Alerts via Rootly Source: https://www.checklyhq.com/docs/integrations/incident-management/rootly Configure Rootly integration to receive real-time alerts from Checkly monitors Checkly integrates with [Rootly](https://rootly.com/) and can deliver failure, degradation, and recovery messages to any Rootly service. More specifically, Checkly will: * Open new incidents when a check fails. * Close incidents automatically when a failing check recovers. * Alert when SSL certificates are about to expire. 1. Log in to Rootly and navigate to **Alerts > Alert Sources**, click **Add Alert Source** and select "Checkly"." 2. Take note of the following: * The **Secret** that Rootly.io provides you with. * The **type** you want to target: either **Service**, **Group** or **Escalation Policy**. * The **ID** of the target. Now head over to [Checkly > Alerts](https://app.checklyhq.com/alerts/settings), click **Add more channels** and select **Rootly**. 3. Copy and paste in the API key, select the type and paste in the ID. 4. Hit **Test Rootly alert** to verify the connection. On the Incident.io side, you should now see a test alert pop up! 5. Now you are all set up! When an alert is triggered, an incident will be created in Rootly. You now have successfully integrated Checkly with Rootly. # Send Alerts via Spike Source: https://www.checklyhq.com/docs/integrations/incident-management/spike Configure Spike integration to receive real-time alerts from Checkly monitors Checkly integrates with [Spike.sh](https://spike.sh) and can deliver failure, degradation, and recovery messages to any service in your Spike.sh dashboard. 1. First, add a new **Integration**. Log in to Spike.sh and go to "Integrations". setup checkly spikesh step 1

2. Select "New Integration", and then choose **Checkly** in the list of available Integrations. Press "Next" to continue on to set it up. setup checkly spikesh step 2

3. On the "Add Integration" page, you can fill in the details of your Checkly integration, like its name, the Spike.sh service to which it should be tied, escalation policy, and more. When you're done, hit **Add integration**. setup checkly spikesh step 3

4. You will now be taken to the Integration overview screen where you can view any activity being generated by this integration. The important part for us is gathering the Webhook URL. So click on **Copy Webhook** to copy the Webhook URL to your clipboard. setup checkly spikesh step 4

5. Next, log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find Spike.sh on the list, and click "Add channel". setup checkly spikesh step 5

6. Give the alert channel a name and **paste the Webhook URL** into the URL field. You can now also tweak which alerts you want to be notified of and which checks or check groups should be attached to your Spike.sh service. setup checkly spikesh step 6

2. Add the **REST generic** integration and click "Enable Integration" on the next screen. Now **copy the URL**. setup checkly splunk integration step 2

Notice that you need to **append a routing key**. This should be the routing key you associated with a group e.g. "database" or "product team" For more details, [see the Splunk / VictorOps knowledge base](https://help.victorops.com/knowledge-base/routing-keys/) 3. Log in to Checkly and navigate to [Alert Settings](https://app.checklyhq.com/alert-settings/). Click the "Add more channels" button, find Splunk On-Call on the list, and click "Add channel". setup checkly splunk integration step 3

4. Give the alert channel a name and **paste the URL** in the dedicated URL input field. You can now also tweak which alerts you want to be notified of and which checks or check groups should be subscribed to this channel. setup checkly splunk integration step 4

2. Next up, navigate to the Alert Settings tab on Checkly and add a new channel. checkly create alert

3. Select `Webhook`. checkly create webhook for statuspage

4. Configuring the webhook, enter `POST` as method and `https://api.statuspage.io/v1/pages//incidents` as URL. The body needs to be configured as shown in StatusPage's [official documentation](https://developer.statuspage.io/#operation/postPagesPageIdIncidents). An example could be: ```json theme={null} { "incident": { "name": "{{ALERT_TITLE}}", "status": "investigating", "impact_override": "critical", "metadata": {}, "deliver_notifications": true, "auto_tweet_at_beginning": false, "auto_tweet_on_completion": false, "auto_tweet_on_creation": false, "auto_tweet_one_hour_before": false, "backfill_date": "string", "backfilled": false, "body": "Something broke :(", "components": {}, "component_ids": [], "scheduled_auto_transition": false } } ``` Where the `name` field will be set to the title of the Checkly alert. See all [available alert variables](/docs/integrations/alerts/webhooks#using-variables). checkly statuspage webhook body and url

5. You will also need to provide your StatusPage API key in the `Authorization` header in the Headers tab, as well as select on which status the webhook should fire ("when a check fails", in this case), and which checks/groups should subscribe to the webhook. Subscribed checks and groups will trigger the webhook and open an incident. checkly webhook authorization header for statuspage

checkly webhook authorization header for statuspage

You are done! When a subscribed check fails, an incident will be opened on StatusPage: checkly statuspage integration end result

Note that currently, automatically resolving existing incidents on check recovery is not supported. Congratulations! You have successfully integrated Checkly with StatusPage! # Export Metrics & Data via Coralogix Source: https://www.checklyhq.com/docs/integrations/observability/coralogix Deep integration with Coralogix for Open Telemetry tracing, RUM monitoring, and alert notifications Checkly deeply integrates with [Coralogix](https://coralogix.com/) in three ways: 1. Checkly can automatically instrument all check executions with **Open Telemetry tracing** and send the data to Coralogix as traces. 2. Checkly can automatically instrument your Browser checks with **Coralogix RUM (Real User Monitoring)**. 3. Checkly can send **alert notifications** when checks pass and/or fail to **Coralogix Logs**. ## Open Telemetry tracing integration To set up the Open Telemetry tracing integration, first enable the Coralogix integration in your Checkly account in the [integrations section](https://app.checklyhq.com/settings/account/integrations). coralogix otel integration

1. Pick the region of your tenant and enter the team name. We use this to determine the correct trace ingest endpoint for Coralogix. 2. Create a "Send Your Data" API key in your Coralogix account, you can find these in the **Data Flow > API Keys** section of your Coralogix dashboard. Copy and paste the API key. The integration is now active and Checkly will start doing two things 1. Send a trace for every check run to Coralogix. 2. Instrument all HTTP requests generated by the check with the following headers: | Header | Value | Description | | ------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `traceparent` | `{version}-{trace_id}-{span_id}-{trace_flags}` | A trace parent ID in [the canonical w3c format](https://www.w3.org/TR/trace-context/#traceparent-header-field-values) | | `tracestate` | `checkly=true` | A vendor specific header so you can recognize traces originating from Checkly | coralogix checkly traces in coralogix

Moreover, it will show a **trace ID** and a **permalink** to the recorded trace in the check result page, so you can easily correlate the check run with the trace in Coralogix. coralogix link in checkly result

Each trace will have the following metadata added to it as **tags**: | Tag | Description | | -------------------------- | ------------------------------------------------------------- | | `checkly.account.id` | The ID of the Checkly account the check belongs to. | | `checkly.check.id` | The ID of the Checkly check that generated the trace. | | `checkly.check.name` | The name of the Checkly check that generated the trace. | | `checkly.check.type` | The type of the Checkly check that generated the trace. | | `checkly.check.runtime` | The runtime ID of the Checkly check that generated the trace. | | `checkly.check.session.id` | The unique ID of the check session the check belongs to. | | `checkly.check.result.id` | The unique result ID of the check run. | | `checkly.check.location` | The location ID of where the check was run, i.e. `us-east-1`. | Using these tags, you can create filters, dashboards and slice & dice your Checkly data as your runs are executed and ingested into Coralogix as traces. If you are using a different trace header than the standard `traceparent`, you can configure this header in the *custom trace header* field in the integration settings. ### What's included * Traces for every check execution * HTTP request instrumentation with W3C headers * Trace correlation metadata Check **metrics** (response times, availability, etc.) are not sent through this integration. To send these metrics to Coralogix, scrape them via the [Prometheus integration](https://www.checklyhq.com/docs/integrations/observability/prometheus-v2/) and forward to Coralogix using an OpenTelemetry Collector. ## RUM integration In addition to the Open Telemetry tracing, Checkly can also send RUM data to Coralogix for all Browser checks. This enables tracking your Playwright-powered checks to be recorded as (semi) real users, so you can compare and contrast the performance between synthetic and real users. coralogix rum integration

1. Click **Enable Coralogix RUM** and pick the region of your tenant. 2. Set the global name of the application you will be targeting with your checks, e.g. "Acme webshop". 3. Create a new RUM integration in your Coralogix account via **Data Flow > Integrations > Real User Monitoring** rum api key area

4. You will need to create an **Integration Key** and a **SourceMap Upload Key**. Copy and paste the integration key into Checkly. > Only the integration key is needed for the Checkly integration. 5. As Checkly automatically injects the Coralogix RUM SDK to each browser check run, uploading and setting up the SourceMap is not required. You can **skip** the **Install & setup Browser SDK Package** step and finish the set up there. If you are targeting multiple applications with your checks, you can override the global application name by adding the environment variable `CORALOGIX_RUM_APP_NAME` to your Browser check. You can find the Browser check specific environment variables under **Settings** > **Environment variables** in the Browser check editor. After enabling the RUM integration, Checkly will start sending RUM data to Coralogix for all Browser checks. > Note that you do not need to install the RUM SDK in your application, Checkly will automatically inject the necessary > JavaScript snippet into your Browser checks. However, if you have installed the SDK we will use the existing installation and not inject the snippet. ## Alert Channel → Coralogix Logs integration You can set up Coralogix as an alert channel destination, so Coralogix receives any alert notifications you have set up as logs. Note that you can use this integration separately from the other two integrations mentioned above. 1. In Checkly, navigate to [alert settings](https://app.checklyhq.com/alerts/settings/). Click the **Add more channels** button, find Coralogix on the list, and click **Add channel**. coralogix alert channels

2. Create a "Send Your Data" API key in your Coralogix account, you can find these in the **Data Flow > API Keys** section of your Coralogix dashboard. Copy and paste the API key. 3. Give your channel a name, and provide the correct Coralogix Logs API endpoint for the appropriate region, e.g. `https://ingress.eu2.coralogix.com/logs/v1/singles` Make sure that the right checks are subscribing to the channel and that the **Send when** rules are correctly set, then hit **Save Coralogix Webhook**. coralogix integration step 4

# Send Alerts via Prometheus Source: https://www.checklyhq.com/docs/integrations/observability/prometheus Configure Prometheus integration to receive real-time alerts from Checkly monitors This page describes the v1 version of the Prometheus exporter. If you are just starting with our Prometheus integration, we recommend using our [v2 version](/docs/integrations/observability/prometheus-v2). Checkly exposes the following metrics in a Prometheus-compatible format. | Metric | Description | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `checkly_check_status` | Whether a check is passing or failing. `1` means the check is passing, `0` means the check is failing. | | `checkly_check_degraded_status` | Whether a check is degraded. `1` means the check is not-degraded, `0` means the check is degraded. | | `checkly_check_result` | The last collected response time for a check in a specific region. This means you get one `checkly_check_result` stanza for each region the check is configured to run in. | | `checkly_private_location_queue_size` | The number of check runs scheduled to a private location and waiting to be executed. In particular, this metric reports the maximum count of scheduled check runs over the past 10 minutes. | | `checkly_private_location_oldest_scheduled_check_run` | The age in seconds of the oldest scheduled job in the private location's queue. In particular, this metric reports the maximum age from the past 10 minutes. | | `checkly_private_location_agent_count` | The number of connected Checkly Agents connected for a private location. An Agent is considered as connected if it's communicated with the Checkly infrastructure in the past 10 minutes. | Each `checkly_check` metric has the following labels: * `check_name`, the name of your check. * `check_type`, either `api` or `browser`. * `tags`, this check's tags. > You can set `key:value` tags in your checks/groups and they will be exported as custom labels in Prometheus. For instance the tag `env:production` will be exposed as a custom label `env="production"`. You can disable this by adding the query param `disableTagParsing=true`. The `checkly_private_location` metrics contain the labels: * `private_location_name`, the name of the private location. * `private_location_slug_name`, the private location's human readable unique identifier. * `private_location_id`, the private location's UUID. > If a private location has no check runs for six hours, it will be considered inactive and `checkly_private_location` metrics won't be reported for it. Here is an example: ```bash theme={null} # HELP checkly_check_status The status of the last check. 1 is passing, 0 is failing # TYPE checkly_check_status gauge checkly_check_status{check_name="Customer API",check_type="api",muted="false",activated="true",tags="alerts,public"} 1 checkly_check_status{check_name="Email login",check_type="browser",muted="false",activated="true",tags="auth,browser-checks,public"} 0 # HELP checkly_check_degraded_status The degraded status of the last check. 1 is not-degraded, 0 is degraded # TYPE checkly_check_degraded_status gauge checkly_check_degraded_status{check_name="Customer API",check_type="api",muted="false",activated="true",tags="alerts,public"} 0 checkly_check_degraded_status{check_name="Email login",check_type="browser",muted="false",activated="true",tags="auth,browser-checks,public"} 1 # HELP checkly_check_result The response time of the last check per region. # TYPE checkly_check_result gauge checkly_check_result{check_name="Customer API",check_type="api",region="ap-northeast-2",tags="alerts,public"} 1168 checkly_check_result{check_name="Customer API",check_type="api",region="ap-southeast-1",tags="alerts,public"} 932 checkly_check_result{check_name="Customer API",check_type="api",region="ca-central-1",tags="alerts,public"} 424 checkly_check_result{check_name="Customer API",check_type="api",region="eu-west-2",tags="alerts,public"} 138 checkly_check_result{check_name="Customer API",check_type="api",region="us-east-2",tags="alerts,public"} 432 checkly_check_result{check_name="Email login",check_type="browser",region="ap-south-1",tags="auth,browser-checks,public"} 10174 # HELP checkly_private_location_queue_size The number of check runs scheduled to the private location and waiting to be executed. # TYPE checkly_private_location_queue_size gauge checkly_private_location_queue_size{private_location_name="Internal CI",private_location_slug_name="internal-ci",private_location_id="cac52f2d-8b8c-4ca5-9711-1836be02eda4"} 0 # HELP checkly_private_location_oldest_scheduled_check_run The age in seconds of the oldest check run job scheduled to the private location queue. # TYPE checkly_private_location_oldest_scheduled_check_run gauge checkly_private_location_oldest_scheduled_check_run{private_location_name="Internal CI",private_location_slug_name="internal-ci",private_location_id="cac52f2d-8b8c-4ca5-9711-1836be02eda4"} 0 # HELP checkly_private_location_agent_count The number of agents connected for the private location. # TYPE checkly_private_location_agent_count gauge checkly_private_location_agent_count{private_location_name="Internal CI",private_location_slug_name="internal-ci",private_location_id="cac52f2d-8b8c-4ca5-9711-1836be02eda4"} 1 ``` Notice that: * The check for "Email login" is failing. * The check for "Customer API" is degraded, but not failing. * The `checkly_check_status` metric has `muted` and `activated` labels, reflecting if a check is sending out alerts or is actually running. * The `checkly_check_result` metric has a `region` label. * The private location "Internal CI" has one Checkly Agent connected. From `checkly_private_location_queue_size` and `checkly_private_location_oldest_scheduled_check_run`, we see that there's no backlog of check run jobs. Activating this integration is simple. 1. Navigate to the integrations tab on the account screen and click the 'Create Prometheus endpoint' button. Prometheus integration step 1

2. We directly create an endpoint for you and provide its URL and the required Bearer token. Prometheus integration step 2

3. Create a new job in your Prometheus `prometheus.yml` config and set up a scraping interval. We recommend an interval between 30 seconds and 60 seconds. Add the URL (divided into `metrics_path`, `scheme` and `target`) and `bearer_token`. Here is an example ```yaml theme={null} # prometheus.yml - job_name: 'checkly' scrape_interval: 30s metrics_path: '/accounts/993adb-8ac6-3432-9e80-cb43437bf263/prometheus/metrics' bearer_token: 'lSAYpOoLtdAa7ajasoNNS234' scheme: https static_configs: - targets: ['api.checklyhq.com'] ``` Now restart Prometheus and you should see metrics coming in. 💡 Check out our [blog post on using this integration with Prometheus and Grafana](https://blog.checklyhq.com/monitoring-website-performance-with-checkly-prometheus-grafana/) for some cool real-life applications. # Exporting Metrics & Data via Prometheus V2 Source: https://www.checklyhq.com/docs/integrations/observability/prometheus-v2 Export Checkly monitoring data to Prometheus for advanced metrics analysis and Grafana dashboards This page describes a new V2 version of the Prometheus exporter. For information about the old Prometheus exporter, see the [Prometheus V1 docs](/docs/integrations/observability/prometheus). If you are using [Prometheus](https://prometheus.io/) for monitoring and the popular [Grafana](https://grafana.com/) stack for dashboarding, you can expose Checkly's core metrics on a dedicated, secured endpoint. ## Activation Activating this integration is simple. 1. Navigate to the integrations tab on the account screen and click the 'Create Prometheus endpoint' button. Prometheus integration step 1

2. We directly create an endpoint for you and provide its URL and the required Bearer token. Prometheus integration step 2

3. Create a new job in your Prometheus `prometheus.yml` config and set up a scraping interval. Use a 60-second scrape interval and enable compression. Add the URL (divided into `metrics_path`, `scheme` and `target`) and `bearer_token`. Here is an example ```yaml theme={null} # prometheus.yml - job_name: 'checkly' scrape_interval: 60s scrape_timeout: 30s enable_compression: true metrics_path: '/accounts//v2/prometheus/metrics' bearer_token: '' scheme: https static_configs: - targets: ['api.checklyhq.com'] ``` Now restart Prometheus and you should see metrics coming in. The Prometheus metrics endpoint has a rate limit of 50 requests per minute. The responses from this endpoint are cached for 60 seconds. Any request made to this endpoint within 60 seconds of the initial request will receive the cached response. Use a scrape interval of 60 seconds. Shorter intervals fetch the same cached data. If you run multiple Prometheus instances, each instance scrapes independently. Align their scrape intervals where possible. Keep `enable_compression` set to `true` to reduce response size and make scrapes faster. Prometheus supports this setting in version 2.49 and later. If your account has many checks, use a longer `scrape_timeout`, such as 30 seconds, to give Prometheus enough time to scrape all metrics. ## Check Metrics The Prometheus exporter exposes several metrics you can use to monitor the status of your checks, as well as to inspect detailed information such as [Web Vitals](/docs/detect/synthetic-monitoring/browser-checks/performance-metrics). The following metrics are available to monitor checks: | Metric | Type | Description | | -------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `checkly_check_status` | Gauge | Indicates whether a given check is currently passing, degraded, or failing. | | `checkly_check_result_total` | Counter | The number of passing, degraded, and failing check results. | | `checkly_browser_check_web_vitals_seconds` | Histogram | The [Web Vitals](/docs/detect/synthetic-monitoring/browser-checks/performance-metrics) timings. | | `checkly_browser_check_duration_seconds` | Histogram | The total check duration. This includes all pages visited and any waits. | | `checkly_browser_check_errors` | Histogram | The errors encountered during a full browser session. | | `checkly_api_check_timing_seconds` | Histogram | The response time for the API request, as well as the duration of the different phases. | | `checkly_url_monitor_timing_seconds` | Histogram | The response time for the HTTP request, as well as the duration of the different phases. | | `checkly_tcp_monitor_timing_seconds` | Histogram | The response time for the TCP request, as well as the duration of the different phases. | | `checkly_tcp_check_timing_seconds` | Histogram | **Deprecated:** use `checkly_tcp_monitor_timing_seconds` instead. Emits identical values. | | `checkly_icmp_monitor_timing_seconds` | Histogram | The timing phases of the ICMP monitor, including DNS resolution and latency measurements (avg, min, max). | | `checkly_icmp_monitor_packet_loss_ratio` | Gauge | The average packet loss ratio (0-1) for the ICMP monitor. | | `checkly_dns_monitor_timing_seconds` | Histogram | The total duration of the DNS monitor. | | `checkly_multistep_check_duration_seconds` | Histogram | The total check duration. This includes all requests done and any waits. | | `checkly_heartbeat_last_success_timestamp_seconds` | Gauge | The Unix timestamp of the last successful heartbeat ping. A value of `0` means no successful ping has ever been received. | | `checkly_heartbeat_expected_interval_seconds` | Gauge | The configured maximum interval between heartbeat pings, in seconds. Derived from the heartbeat monitor's period and period unit settings. | | `checkly_time_to_ssl_expiry_seconds` | Gauge | The amount of time remaining before the SSL certificate of the monitored domain expires. See the [SSL certificate expiration docs](/docs/communicate/alerts/ssl-expiration) for more information on monitoring SSL certificates with checks. | The `checkly_check_status` and `checkly_check_result_total` metrics contain a `status` label with values `passing`, `failing`, and `degraded`. The `checkly_check_status` gauge is `1` when the check has the status indicated by the `status` label and is `0` otherwise. For example, if a check is passing the result will be: ```bash theme={null} checkly_check_status{name="Passing Browser Check",status="passing"} 1 checkly_check_status{name="Passing Browser Check",status="failing"} 0 checkly_check_status{name="Passing Browser Check",status="degraded"} 0 ``` `checkly_check_status` can be useful for viewing the current status of a check, whereas `checkly_check_result_total` can be useful for calculating overall statistics. The metrics `checkly_browser_check_web_vitals_seconds`, `checkly_browser_check_errors`, `checkly_api_check_timing_seconds`, `checkly_tcp_monitor_timing_seconds`, `checkly_icmp_monitor_timing_seconds`, and `checkly_dns_monitor_timing_seconds` contain a `type` label. This label indicates the different Web Vitals, error types, and timing phases being measured. For `checkly_tcp_monitor_timing_seconds`, the `type` label has the following values: `dns`, `connection`, `data`, and `total`. For `checkly_icmp_monitor_timing_seconds`, the `type` label has the following values: `dns` (DNS resolution time), `latency_avg`, `latency_min`, and `latency_max`. For `checkly_dns_monitor_timing_seconds`, the `type` label has the value `total`. `checkly_time_to_ssl_expiry_seconds` contains a `domain` label giving the domain of the monitored SSL certificate. In addition, the check metrics all contain the following labels: | Label | Description | | ------------ | ---------------------------------------------------------------------------------------------------------- | | `name` | The name of the check. | | `check_id` | The unique UUID of the check. | | `check_type` | The type of check: `api`, `browser`, `multi_step`, `playwright`, `heartbeat`, `tcp`, `icmp`, `dns`, `url`. | | `muted` | Whether the check is muted, configured to not send alerts. | | `activated` | Whether the check is activated. Deactivated checks aren't be run. | | `group` | The name of the check group. | | `tags` | The tags of the check. | > You can set `key:value` tags in your checks and groups and they will be exported as custom labels in Prometheus. For instance the tag `env:production` will be exposed as a custom label `env="production"`. You can disable this by adding the query param `disableTagParsing=true`. Please note that Prometheus label names may only contain ASCII letters, numbers, as well as underscores (see the official [docs](https://prometheus.io/docs/concepts/data_model/)). Tags containing other characters in the label name will be sanitized. > The counter and histogram metrics are reset every hour. These resets can be handled in Prometheus by using the [rate](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) or [increase](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase) functions. ### Check Run Location Label To avoid creating a high volume of metrics, by default the metrics don't include a label for the check run location. It is possible to enable this by adding the query param `locationLabelEnabled=true` to your API request. This will add a `location` label giving the location where the checks ran. Since check status and SSL days remaining is only tracked on a per-check basis rather than by location, `checkly_check_status` and `checkly_time_to_ssl_expiry_seconds` do not have the `location` label included. Heartbeat metrics (`checkly_heartbeat_last_success_timestamp_seconds`, `checkly_heartbeat_expected_interval_seconds`, and heartbeat entries in `checkly_check_result_total`) also do not have a `location` label, since heartbeat monitors receive pings rather than running from specific locations. All other [check metrics](#check-metrics) will have the `location` label added. Here is an example for how to set this in your `prometheus.yml` config: ```yaml theme={null} # prometheus.yml - job_name: 'checkly' scrape_interval: 60s scrape_timeout: 30s enable_compression: true metrics_path: '/accounts//v2/prometheus/metrics' bearer_token: '' scheme: https static_configs: - targets: ['api.checklyhq.com'] params: locationLabelEnabled: ['true'] ``` ### Check Retry Attempts By default, [retry attempts](/docs/communicate/alerts/retries/) are not included in the reported metrics. For example, if a check is failing on the initial attempt and passing on the retry, `checkly_check_result_total` and other metrics will only report the final passing check result. It is possible to include retry attempts by adding the query param `includeRetryAttempts=true` to your API request. Metrics reporting information on check results will then include retry attempts. An exception to this behaviour is `checkly_check_status`, which reports whether the check is currently in a passing, degraded, or failing state. Since retry attempts don't affect the checks state and only the final check result determines the state, this metric ignores retry attempts regardless of the `includeRetryAttempts` setting. For example, if a check is failing on retry attempts and passing on the retry, `checkly_check_status` will report that the check is passing. Here is an example for how to configure `includeRetryAttempts` in your `prometheus.yml` config: ```yaml theme={null} # prometheus.yml - job_name: 'checkly' scrape_interval: 60s scrape_timeout: 30s enable_compression: true metrics_path: '/accounts//v2/prometheus/metrics' bearer_token: '' scheme: https static_configs: - targets: ['api.checklyhq.com'] params: includeRetryAttempts: ['true'] ``` ### PromQL Examples This section contains a few PromQL queries that you can use to start working with the Prometheus data. #### Currently failing checks To graph whether checks are passing or failing, use the query: ```bash theme={null} checkly_check_status{status="passing"} ``` Passing checks will have the value `1` while failing and degraded checks will have the value `0`. This can be used to build a [Grafana table](https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/table/) of currently failing checks. #### Failure percentage To calculate the percentage of check runs that failed in the last 24 hours, use: ```bash theme={null} increase(checkly_check_result_total{status="failing"}[24h]) / ignoring(status) sum without (status) (increase(checkly_check_result_total[24h])) ``` #### Histogram averages The different histogram metrics can all be used to compute averages. For example, query the average web vitals times for a check using: ```bash theme={null} sum by(type) (rate(checkly_browser_check_web_vitals_seconds_sum{name="Check Name"}[30m])) / sum by(type) (rate(checkly_browser_check_web_vitals_seconds_count{name="Check Name"}[30m])) ``` ## Heartbeat Metrics [Heartbeat monitors](/docs/detect/uptime-monitoring/heartbeat-monitors/overview) are passive checks that wait for your scheduled jobs and automated processes to send a ping. The Prometheus exporter includes heartbeat checks in the standard `checkly_check_status` and `checkly_check_result_total` metrics, plus two heartbeat-specific gauges designed for [dead man's switch](https://en.wikipedia.org/wiki/Dead_man%27s_switch) alerting. * **`checkly_heartbeat_last_success_timestamp_seconds`** — The Unix timestamp of the last successful ping, searched across the entire event history (no time window). A value of `0` means no successful ping has ever been received — this immediately triggers a dead man's switch alert since `time() - 0` produces a very large value. * **`checkly_heartbeat_expected_interval_seconds`** — The maximum expected time between pings, derived from the heartbeat monitor's configured period and period unit (e.g., `5 minutes` = `300` seconds). Use this as a threshold for alerting. ### How heartbeats map to `checkly_check_result_total` Heartbeat checks contribute to `checkly_check_result_total` using a 1-hour count window, the same as other check types. The `status` label values map to heartbeat event states as follows: | `status` label | Heartbeat states counted | | -------------- | ------------------------------------------------------------ | | `passing` | `RECEIVED`, `EARLY`, `GRACE`, `LATE` — any ping that arrived | | `failing` | `FAILING` — no ping received before the grace period expired | | `degraded` | Always `0` — heartbeat monitors do not have a degraded state | ### Dead Man's Switch PromQL Examples #### Alert when a heartbeat ping is overdue Compare the current time against the last success timestamp. If the difference exceeds the expected interval, the job has missed its window: ```bash theme={null} time() - checkly_heartbeat_last_success_timestamp_seconds > checkly_heartbeat_expected_interval_seconds ``` #### Alert with a safety margin Add a grace period multiplier (e.g., 1.5x) to avoid false alarms from minor delays: ```bash theme={null} time() - checkly_heartbeat_last_success_timestamp_seconds > 1.5 * checkly_heartbeat_expected_interval_seconds ``` #### Grafana table of heartbeat health Show how long ago each heartbeat was last seen: ```bash theme={null} time() - checkly_heartbeat_last_success_timestamp_seconds ``` ## Private Location Metrics The Prometheus exporter also contains metrics for monitoring [Private Locations](/docs/platform/private-locations/overview). These metrics can be used to ensure that your Private Locations have enough Checkly Agent instances running to execute all of your checks, and to drive [agent autoscaling](/docs/platform/private-locations/autoscaling). The following metrics are available to monitor Private Locations: | Metric | Type | Description | | ----------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `checkly_private_location_queue_size` | Gauge | The number of check runs scheduled to the Private Location and waiting to be executed. A high value indicates that checks are becoming backlogged and that you may need to [scale your Checkly Agents](/docs/platform/private-locations/scaling-redundancy). | | `checkly_private_location_oldest_scheduled_check_run` | Gauge | The age in seconds of the oldest check run job scheduled to the Private Location queue. A high value indicates that checks are becoming backlogged. | | `checkly_private_location_queue_duration_ms` | Gauge | The scheduling delay in milliseconds for check runs routed through a Private Location, by queue and `stat`. Derived from the last 2 minutes of check run data. Used to evaluate whether sufficient agent capacity is provisioned for the Private Location. | | `checkly_private_location_agent_count` | Gauge | The number of agents connected for the Private Location. | | `checkly_private_location_check_runs` | Gauge | The count of check runs in the Private Location, broken down by lifecycle state via the `state` label. Used as the signal for [autoscaling Checkly Agents](/docs/platform/private-locations/autoscaling). | The Private Location metrics all contain the following labels: | Label | Description | | ---------------------------- | -------------------------------------------------------- | | `private_location_name` | the name of the Private Location. | | `private_location_slug_name` | the Private Location's human readable unique identifier. | | `private_location_id` | the Private Location's UUID. | `checkly_private_location_check_runs` additionally carries a `state` label. The values relevant for autoscaling are: * `queued` — the check run has been scheduled but not yet picked up by an agent. * `inflight` — the check run is currently being executed by an agent. `checkly_private_location_queue_duration_ms` additionally carries two labels: * `stat` — the statistical aggregation over the 2-minute window. Values: `avg`, `p90`, `p99`. * `queue` — the internal routing queue the check run was scheduled to. Values: `uptime`, `main`. This gauge is aggregated on a \~1 minute interval, so check runs that start and finish within that window may be excluded. # Anatomy of a Status Page | Checkly Guide Source: https://www.checklyhq.com/docs/learn/incidents/anatomy-of-a-status-page Discover what makes a status page users trust - automated monitoring, clear visuals, service groupings, and more. When I first started in the tech industry, a new idea was on the scene for your service’s status page: just embed the company’s Twitter (now X) account on a page. Since we were all using Twitter constantly, it was the best place to check for updates about our service. These days, our expectations for a critical service’s status page are a bit higher than the occasional social media post. This article helps define what users expect in a status page, and the technical requirements. ## What users expect from a status page A status page helps users know if your applications or services are working. Here’s what users expect from a good status page: ### 1. Clear Status Updates * Show if services are **up, down, or having issues**. * Use simple labels (like "Operational," "Degraded," or "Outage"). ### 2. Externally Hosted * The page should be on a separate system (not your main website). This way, users can check it even if your site is down. ### 3. Subscribe Function * Let users **subscribe** to updates via email, SMS, or RSS. * Notify them when status changes. ### 4. Easy for Everyone to Use * **Regular users** should understand it easily. * **Application owners** should be able to manually mark a service as down if needed. ### 5. Multiple Sections * Support different **audiences** (e.g., one page for customers, another for internal teams). * Let users **subscribe only to the pages they care about**. A good status page keeps users informed, reduces frustration, and builds trust. Make sure yours is simple, reliable, and easy to follow. Layers of Incident Response

## Technical requirements for a good status page The basic technical requirements (fast loading pages, manual edits permitted, etc) aren’t worth mentioning, but the following features will require some ‘lift’ on any DIY solution, and aren’t offered by every SaaS status page tool. **1. Automated monitoring** - no matter how slick our status page, if we’re relying on users to report most outages, our status page is worse than useless. Implicit in the requirements above is that most outages and incidents will be detected automatically and added to a status page. The best way to determine your service status automatically is through [synthetic monitoring](/docs/learn/monitoring/transaction-monitoring), ideally using [Playwright](https://www.checklyhq.com/learn/playwright/) to simulate user paths in detail. **2. Independent hosting** - ideally both your monitoring and the status page will be hosted away from your service’s cloud. Even AWS are vulnerable to failures where the monitor (and its notifications) failed in the same outage that took down the service, adding significantly to [time to repair](https://www.checklyhq.com/learn/incidents/mttr-challenges/). **3. Clear infographics** - a status page should be able to take complex historical data and turn it into a clear history for the viewer. Some kind of graphing library should represent high level status data visually. **4. Incidents should be organized into services** - for Checkly’s Status Pages, all alerts are [classified ahead of time into services](/docs/communicate/status-pages/overview#services). We don’t want to present our product as a monolith, nor should we print inscrutable internal labels. With this organization, we can present sections on a single status page, or multiple separate status pages with groupings of services. **5. Subscription service** - Users should subscribe to only the pages they care about. ## Conclusions: for status pages brevity, is wit In short, a good status page keeps users informed and builds trust. It must be clear, reliable, and easy to use. Users want quick updates, simple labels, and the ability to subscribe. On the tech side, it needs automated monitoring, independent hosting, and clear visuals. By meeting these needs, your status page will help users stay updated and reduce frustration. **Further reading:** [A guide to Checkly’s Status Pages](/docs/communicate/status-pages/overview) # Incident Response Automation - From Alerts to Action Source: https://www.checklyhq.com/docs/learn/incidents/automation-incident-response Discover how to layer semi‑automated workflows, automated scripts, and safety checks into your playbooks for faster recovery. While human judgment remains essential, automation can significantly reduce response times, minimize human error, and streamline workflows. This guide explores how developers can integrate automation into incident playbooks effectively, balancing speed with safety. Often SaaS offerings will promise to ‘handle outages fully automatically’ but that’s generally not the reality. Why can’t we automate incident response, and what levels of automation make sense for your team? ### **1. Automation Spectrum in Incident Response** Automation in incident playbooks exists on a spectrum: * **Manual Execution** (Engineers follow step-by-step instructions) * **Semi-Automated** (Engineers trigger scripts/commands via ChatOps or runbooks) * **Fully Automated** (Self-healing systems execute remediation without human intervention) Most organizations lean toward **semi-automation** due to risk tolerance, but the right approach depends on: * **System complexity** (e.g., cloud-native vs. legacy infrastructure) * **Failure domain isolation** (Can automation safely fix this without cascading failures?) * **Regulatory/compliance requirements** (Some industries require human approval for changes.) ## **2. Key Automation Use Cases in Playbooks** **A. Alert Enrichment & Context Gathering** Automation can pre-fetch diagnostic data before engineers engage: * **PagerDuty workflows** that query monitoring systems (Prometheus, Datadog) and attach metrics to incidents. * **ChatOps bots** that dump recent logs, deployment history, or topology maps into the incident channel. Example: ```ts theme={null} // PagerDuty automation script to fetch error rates async function enrichIncident(incidentId: string): Promise { const metrics = await datadogApi.query(`system.error_rate by ${service}`); await pagerduty.updateIncident(incidentId, { context: metrics }); } ``` One example of alert enrichment is the large amount of easily readable information offered in an alert from Checkly. At the high level, failures are shown along with failed attempts that didn’t trigger an alert because of a successful retry: **B. Safe, Repeatable Remediation Steps** Automate actions with well-understood outcomes: * **Restarting a stuck service** (e.g., `kubectl rollout restart deployment/{service}`) * **Scaling up resources** (e.g., AWS Lambda concurrency increase) * **Blocking a malicious IP** (e.g., via Cloudflare API) Example (ChatOps): ``` !runbook restart_service --env=prod --service=payment-gateway ``` *→ Bot executes pre-approved Ansible playbook and reports back.* **C. Automated Triage & Escalation** * **Classify incidents** for example"Is this a database issue or network flakiness?") * **Route to the right team** based on symptoms (e.g., SRE vs. Data Engineering) * **Follow availability and escalation rules** this is the kind of thing that is closely associated with PagerDuty: notifying the person who’s currently on-call, and escalating as needed if that person doesn’t respond ## **3. Tools & Integration Patterns** | **Tool** | **Use Case** | **Example Integration** | | ----------------------------------- | -------------------------------------- | ---------------------------------------------- | | **PagerDuty** | Orchestrating workflows, notifications | Auto-trigger AWS Lambda on incident creation | | **ChatOps (Slack/MS Teams)** | Human-in-the-loop automation | Bot executes `kubectl` commands after approval | | **Runbook Tools (Confluence, Git)** | Documentation-as-code | Markdown with embedded Terraform snippets | | **Ansible/Chef** | Safe, idempotent remediation | Rollback to last known good config | | **Serverless (AWS Lambda)** | Lightweight automation hooks | Auto-mitigate S3 bucket throttling | ## **4. Security & Guardrails** Automation introduces risks—**fail-safes are critical**: * **Approval workflows** (e.g., "Execute repair? ✅/❌" in Slack) * **Dry-run modes** ("What would this script do?") * **Blast radius control** (Limit parallelism, region scoping) * **Audit logs** (All actions should be traceable to an incident ID.) Example safety check: ```bash theme={null} if [[ $ENVIRONMENT == "prod" ]]; then echo "Prod changes require manual approval." | slack-notify exit 1 fi ``` ## **5. Cultural & Organizational Factors** * **Start small**: Automate only the most repetitive, low-risk tasks. * **Trust through transparency**: Engineers should see what automation is doing (e.g., ChatOps command logging). * **Post-mortem feedback loops**: Analyze if automation helped or caused issues. A quote from one engineer talking about how automation can get rolled back after errors: > "At one company, we automated DNS failover—until it once failed over unnecessarily. Now it pings the on-call first." ### The final word on Automation in playbooks Automation in incident playbooks isn’t about replacing humans—it’s about **augmenting response efforts** while minimizing toil. By focusing on **enrichment, safe remediation, and guardrails**, developers can build systems that accelerate recovery without sacrificing reliability. **Final tip**: If your automation can’t explain its actions in a post-mortem, it’s not ready for production. # Defining MTTR - what mean time to repair means for your team Source: https://www.checklyhq.com/docs/learn/incidents/defining-mttr Mean Time to Repair (MTTR) is a critical metric in the world of IT operations and engineering, representing the average time it takes to repair a system or service after a failure occurs. Despite its importance, many teams fail to meet their MTTR goals. Mean Time to Repair (MTTR) is a critical metric in the world of IT operations and engineering, representing the average time it takes to repair a system or service after a failure occurs. Despite its importance, many teams fail to meet their MTTR goals. MTTR or its corollary may be essentially to meeting your Service Level Agreements (SLA) with your clients, which in their most basic form are an agreement about the amount of time that your service is available in a given period. If unintentional outages (failures) are taking your service offline frequently, and the outages take a long time to resolve, you won’t meet your SLA. ## Understanding MTTR Originally a term from old school IT and even Operational services, MTTR used to refer to the time it took from a physical device’s breakdown to it its being back in service. For example, when an electric motor would break down, how long before a technician had analyzed the problem, fixed the motor, and it was back driving industrial machinery. MTTR stands for "mean time to repair," but its definition can vary depending on whom you ask. Some interpret it as the time it takes to restore a system to a functional state, while others see it as the time required to completely fix the underlying issue. This ambiguity makes it difficult to benchmark MTTR across organizations, but it doesn’t explain why so many companies fail to meet their own internal targets. The confusion around MTTR is compounded by the fact that it is sometimes used interchangeably with other terms, such as: * **Mean Time to Respond**: The time it takes to acknowledge and begin addressing an issue. * **Mean Time to Restore**: The time it takes to restore service after an outage. * **Mean Time to Remediate**: The time it takes to fully resolve the root cause of a problem. A timeline of various moments in the incident response cycle that can sometimes be called MTTR

A timeline of various moments in the incident response cycle that can sometimes be called MTTR

Most teams do default to the root meaning, or the original meaning, "mean time to repair," but even within this definition, there is room for interpretation. This lack of clarity can lead to misaligned expectations and ineffective strategies for improving MTTR. ## Calculating MTTR The Mean Time to Repair is just the arithmetic mean (or average) of your time to resolution from all incidents. The timeline diagram above gives an example of how the time to repair is measured from each incident: it starts when the outage first occurred, to when a fix was deployed. If we had five incidents last month, with times to repair looking like the following table: | Incident Number | Time to Repair | | --------------- | -------------- | | 1 | 22 minutes | | 2 | 9 minutes | | 3 | 86 minutes | | 4 | 21 minutes | | 5 | 147 minutes | The mean time to repair would be: (22 + 9 +86 + 21 + 147 minutes) / 5 incidents = (285 minutes) / 5 incidents = **57.2 minutes** per incident Note that mean time to repair is not a full indicator of how you’re meeting your SLA, since you must multiply back out the number of incidents you’ve had. ## How MTTR can fall short of measuring performance It seems a simple thing to say ‘we want the minimum mean time to repair’ for a team and that the resulting optimization of the resolution process will result in better performance for users. As a very general critique, allow me to quote Goodhart’s Law: > Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. When we first proposed MTTR as a measure for how well we were meeting our service level agreement (SLA) for our users, the connection between these statistics seemed irrefutable. But talking to senior Ops engineers you will hear many stories about how, despite improvements in MTTR and other metrics, users still complained of broken SLAs. One example: “We had an enterprise client complain we weren’t meeting our uptime agreement. This was after we’d streamlined the incident response process and put in hard controls for how downtime was measured. The issue it turned out was how slowly we marked downtime as ‘resolved’ on our status page. The client was monitoring that page and using it to show more downtime than we really had.” On the level of an individual conversation that can seem like a simple miscommunication, but take it from the user’s view: they rely on the status page to know when the service is available after an incident. If your team, despite fixing the issue quickly, is taking 20 minutes to remove the downtime status, isn’t that breaking your SLA? ### Disagreements in measuring MTTR The above example is a disagreement about what MTTR is measuring, with one person going by how long a service was unavailable based on backend logging, the other basing off the status page, but there are even more cases where it’s hard to agree on a value. What about geographically specific failures? If a service is unavailable in Germany only, does that [geographic-specific outage](/docs/concepts/locations) count as downtime? What if one feature isn’t working for a week, but other users never notice the issue because they never use that feature? Both these questions ask ‘Does MTTR measure the whole system or just parts?’ and every organization will have its own answer to this question. To refer back to Goodhart’s Law mentioned in the above section, a confounding issue is the use of a general statistic to measure performance and (often) punish failure. When good engineers are facing penalties for missing SLA or hurting MTTR, they can often become very stingy with what counts as downtime. ## The right strategy for measuring MTTR The best way to measure MTTR consistently is to use an automated synthetic monitor that’s agreed on before an incident. If a team is trying to examine an incident post-hoc and define the time points where downtime started and ended, motivated reasoning and multiple definitions will make the statistic less reliable. But if a monitoring service is checking often enough to measure both outages and recoveries in high resolution, this results in a statistic that all sides can agree on. Two additional features of a synthetic monitor should be considered, based on the disagreements about measurement that are so common: * Multiple geographic locations to ensure the service is up for all users * Complex scripting and customization to ensure all core features are working with each check With these tools it’s possible to measure an MTTR that everyone can agree on. Note that to get accurate data, you’ll need to [set the right cadence](https://www.checklyhq.com/blog/check-frequency/) for automated checks. ## Conclusion MTTR is a vital metric for measuring operational performance, but its effectiveness depends on clear definitions, effective tools, and optimized processes. Later articles will discuss the components of a time to repair (time to detect, root cause analysis, and incident response), and propose best practices for meeting SLA. An automated tool that everyone can agree is observing the user experience from multiple locations and with a full simulation of user paths, can help get us a consistent MTTR measurement. [Try Checkly](https://www.checklyhq.com/) to see how our monitoring tools can warn you faster than ever of an outage. # Best Practices for Detecting Downtime | Incident Response Source: https://www.checklyhq.com/docs/learn/incidents/detection-best-practices Discover how to implement multi‑layered monitoring, smart alerts, third‑party checks, failure drills & global probes to catch outages fast and reduce revenue loss Downtime can be costly—leading to lost revenue, frustrated users, and damage to your brand’s reputation. As a developer, ensuring high availability and quickly detecting outages is critical. ## **1. Implement Multi-Layered Monitoring** Relying on a single monitoring method can lead to blind spots. Instead, use a combination of: * **Synthetic Monitoring** (Probes): Simulate user interactions from different locations to detect failures before real users do. * **Application Monitoring**: Gather traces internal from your application, and look for errors or long timespans that indicate trouble. Send logs to a single locality to help you investigate problems after one is detected. * **Infrastructure Monitoring**: Use tools like Prometheus, Grafana, or Nagios to keep an eye on the underlying health of your system. ## **2. Set Up Intelligent Alerting** Avoid "alert fatigue" by ensuring alerts are meaningful and actionable: * **Use Multi-Signal Confirmation**: Trigger alerts only when multiple checks fail (e.g., HTTP status + latency + DNS resolution). For example, Checkly makes it easy to require 2 or more failures before sending alerts in its [retry configuration](https://www.checklyhq.com/docs/communicate/alerts/configuration). * **Prioritize Alerts**: Distinguish between *critical* (full outage) and *warning* (degraded performance). Any time your on-call team gets an alert that doesn’t require action, this should be counted as a false alarm and call for improvements to your alert thresholds. * **Avoid Static Thresholds**: When a service that usually responds in 4.98 seconds starts responding after 5.1 seconds, it’s a negative trend that might call for some investigation. But it certainly does *not* call for a midnight alert to the on-call team. As just one example, Playwright’s assertions about page elements use an [auto-waiting](https://www.checklyhq.com/blog/playwright-auto-waiting/) system that is more flexible than static time thresholds. ## **3. Monitor Third-Party Dependencies** Your application may depend on APIs, CDNs, or cloud providers—any of which can fail. * **Check External Endpoints**: Regularly test third-party services. With [Checkly’s API monitors](/docs/detect/synthetic-monitoring/api-checks/overview), you can send frequent quick checks to external API’s you rely on, and get reliable notifications if something isn’t working, before users report a problem. * **Implement Fallbacks**: Gracefully degrade functionality if a dependency fails. For core dependencies this might go as far as switching to a backup payment provider if your primary service fails. ## **4. Test Failure Scenarios** Proactively simulate outages to ensure your monitoring works: * **Monitor Proactively**: Use synthetic monitoring to send requests to your service. Ideally a [sophisticated automation tool like Playwright](https://www.checklyhq.com/guides/playwright-testing-to-monitoring/) to simulate full user paths. For example on your eCommerce site Playwright could test the process of visiting the site, browsing inventory, adding items to a cart, and checking out. * **Chaos Engineering**: Large teams with dedicated platform engineers can use tools like Chaos Monkey to intentionally break systems and verify detection. * **Run Scheduled Drills**: Periodically trigger false outages to test alerting responsiveness. Note this is only necessary if you aren’t having frequent real incidents! It is useful to schedule a drill whenever the process for incident management changes, so that your team has a chance for a dry run of the procedure. *** ## **5. Ensure Global Visibility** Downtime may only affect certain regions due to DNS, CDN, or cloud provider issues. * **Distributed Monitoring**: Deploy checks across multiple geographic locations. * **Monitor DNS & CDN Health**: Use tools like Checkly to [detect regional outages](/docs/concepts/locations). *** ## **6. Automate Incident Response** When downtime occurs, speed is crucial. * **Auto-Remediation**: For known issues (e.g., a stuck process), automatically restart services. Tools like Kubernetes have automatic remediation as part of their core function. * **Escalation Policies**: Ensure alerts reach the right team (SMS, Slack, PagerDuty). And that incidents are tracked when the team is responding. * **Postmortem Culture**: Analyze outages to improve future detection. *** ## **Conclusion** Detecting downtime quickly requires a proactive, multi-layered approach. By combining synthetic checks, real-user monitoring, intelligent alerting, and global visibility, you can minimize downtime and maintain a reliable user experience. **Further Reading:** * [Reducing Mean Time to Repair](https://www.checklyhq.com/learn/incidents/mttr-challenges/) # How to Use DORA Metrics the Right Way | Checkly Source: https://www.checklyhq.com/docs/learn/incidents/dora-metrics Learn to track and interpret DORA’s four metrics—deployments, lead time, failure rate, and restore time—to improve your software delivery. DORA metrics help teams measure how well they deliver software. DORA stands for DevOps Research and Assessment. The four key metrics are: 1. **Deployment Frequency**: How often you release code. 2. **Lead Time for Changes**: How long it takes to go from code commit to live use. 3. **Change Failure Rate**: How often deployments cause problems. 4. **Time to Restore Service**: How quickly you fix issues when they happen. DORA metrics measurements are inherently subjective to your team and **not a good standard for evaluation between two totally separate teams** working in separate shops. As just one example, if your team’s developers regularly do final integration checks by attempting a canary deployment of a branch, your team will have a much higher change failure rate than a team that does integration tests locally. However this difference won’t indicate a difference in team performance. ## How each DORA metric is measured, and why it matters There’s an interesting variety in DORA metrics. Some are unequivocal in their interpretation (it’s always better to have a lower ‘time to restore service’), while others have no meaning outside context (I could write a script that makes small grammar edits and deploys them, massively boosting ‘deployment frequency’ with no benefit). Some are easy to measure like deployment frequency, others like change failure rate require some thought about *what* is being measured before we even ask why it matters. Here’s all four metrics in detail. ### Deployment Frequency Deployment frequency measures how often a team successfully releases code to production. Frequent deployments often show that a team has a strong CI/CD pipeline and works well together. It helps teams get feedback faster and adapt to changes quickly. Unlike other DORA metrics, higher deployment frequency is better. To make it easier to track, some teams calculate the average time between deployments instead. A higher average time means fewer releases. Measuring deployment frequency is simple. You just need to know when deployments happen. You can then count how often they occur daily, weekly, or monthly. If there are no deployments in a time period, you note that too. More frequent deployments usually mean a faster and more agile team. Top teams (elite performers) deploy multiple times a day, while slower teams might deploy only once a month or less. Deployment frequency is also an object lesson in why **DORA metrics don't translate well for comparison between teams**. If a single team starts deploying code more frequently it does indicate improvement, but just because one team deploys code every hour, and another team only deploys code twice a week, that's not a great indicator of a gap in performance. comic, transcript: 🧑🏻‍💼: Why are your DORA metrics worse than Team A's? 🧑🏽‍💻: Their deployment frequendy is way higher, but.🧑🏻‍💼: ...but? 🧑🏽‍💻: They don't use a CMS. 🧑🏻‍💼: So? 🧑🏽‍💻: So every new blog post requires a git merge. 🧑🏻‍💼: So their DORA metrics are- 🧑🏽‍💻: Apples and oranges, yup.

comic, transcript: 🧑🏻‍💼: Why are your DORA metrics worse than Team A's? 🧑🏽‍💻: Their deployment frequendy is way higher, but.🧑🏻‍💼: ...but? 🧑🏽‍💻: They don't use a CMS. 🧑🏻‍💼: So? 🧑🏽‍💻: So every new blog post requires a git merge. 🧑🏻‍💼: So their DORA metrics are- 🧑🏽‍💻: Apples and oranges, yup.

Factors like underlying code structure and QA practices can cause large differences between teams, so it's best to establish baselines for your team based on past performance. ### Lead Time for Changes Lead time for changes measures how long it takes for code to go from being written to running in production. To calculate it, find the time between when a commit is made and when it’s deployed. Use the middle value (median) of these times over a set period. Shorter lead times mean your team can deliver updates, fixes, or new features faster. This usually shows that your development and deployment processes are working well. To measure lead time, start tracking when a pull request (PR) is created or merged. Then, find out when that code is deployed to production. You’ll need to link the PR to the deployment using an ID, like a tag that connects the two. Once you have the time differences, add them up and divide by the number of changes to get the average. In general, shorter lead times are better. For example, top teams (elite performers) deploy changes in less than a day, while slower teams might take weeks or even months. Faster lead times mean your team can respond quickly to needs and keep improving. ### Time to Restore Services Time to restore services measures how long it takes to fix a service after it fails. The clock stops when the issue is resolved, and the related bug or incident report is closed. A shorter time to restore services means your team handles incidents well and your system is strong. It reduces downtime and keeps users happy. Measuring this metric can be tricky because you need to know exactly when an incident starts and ends. Here are three ways to track it: 1. **Synthetic Monitoring**: Use tools to send regular requests to your service. If it fails, you know the exact downtime. However, it might miss issues where the service seems fine but isn’t working correctly. 2. **Error Logs**: Track errors or exceptions in your code. This can help spot issues, but it might also flag non-critical problems as incidents. 3. **Performance Thresholds**: Watch for unusual changes, like slow response times. Even if the service is technically up, users might feel it’s down if it’s too slow. If you’re not already tracking incidents closely, you might need to start using observability tools or manually note incident times during post-mortems. To calculate the metric, add up the time spent on all incidents and divide by the number of incidents. Faster recovery times are better. Top teams (elite performers) fix issues in less than an hour, while slower teams might take days or even weeks. ### Change Failure Rate Change failure rate measures how often deployments cause problems. It’s the percentage of deployments that fail out of the total number of deployments. A lower change failure rate means your system is reliable and your testing works well. It shows that new changes are less likely to break things. To measure this, you need to count how many deployments fail and compare it to the total number of deployments. However, this can be tricky because not all failures are the same. For example, a long outage might look better than many short ones, even if the short ones are less harmful. Here are three ways to make this metric more accurate: 1. **Standard Rollback Process**: If your team always rolls back failed changes and tags them, you can track failures directly. 2. **Canary Deployments**: Use tools like Argo Rollouts to test changes on a small group before full deployment. Count rollbacks as failures. 3. **Incident Standards**: Set rules for what counts as a failure. For example, only count incidents that last longer than a certain time. Change failure rate can feel less precise than other DORA metrics because it depends on how you define a failure. But it’s still useful for spotting trends. To calculate it, divide the number of failed deployments by the total number of deployments in a set time. A lower rate is better. Top teams (elite performers) keep their failure rate below 15%, while slower teams might see rates as high as 60%. ## DORA matters for developer velocity If your code is easy to test and release, and your development environment is very close to production, you’ll have fewer rollbacks and faster releases. This speed isn’t just about technical skill—it shows your team is agile and better at meeting user needs. Using DORA metrics isn’t just a technical task; it’s a must for platform engineers and team leaders. These metrics give a full picture of your development process, from writing code to deploying it and fixing issues. They show how agile, efficient, and fast your team is. It’s easy to focus only on how much code your team produces, but DORA metrics remind us that the developer experience is just as important. A slow or error-filled deployment process can slow down even the best teams. Improving tools and making the developer experience smoother are key to boosting these metrics. ## Conclusions: Best practices for using DORA metrics These metrics show how fast and reliable your software delivery is. But using them wrong can cause poor evaluation and hurt performance. Here’s how to do it right: 1. **Focus on Improvement, Not Comparison**: Use DORA metrics to find areas to improve, not to compare teams. Every team is different. 2. **Combine Metrics**: Look at all four metrics together. For example, fast deployments are good, but not if they cause many failures. 3. **Understand Context**: Metrics alone don’t tell the whole story. Talk to your team to understand why numbers are high or low. 4. **Avoid Over-Measuring**: Don’t track too many things. Stick to the four DORA metrics to keep it simple. 5. **Use Tools Wisely**: Tools can help track metrics, but don’t rely on them too much. Focus on what the numbers mean for your team. DORA metrics are useful, but only if you use them to learn and improve, not to blame or stress your team. Keep it simple, focus on teamwork, and use the data to make your process better. # Incident Assessment & Severity | Checkly Guide Source: https://www.checklyhq.com/docs/learn/incidents/incident-assessment-severity Not every alert is an incident—and not every incident is equally urgent. How to classify incidents and determine their severity? ## **Incident Assessment & Severity** Not every alert is an incident—and not every incident is equally urgent. That’s where **incident assessment** and **severity classification** come in. Without clear definitions, teams get stuck in limbo: * Should we wake someone up? * Should we inform customers? * Should we prepare a support strategy? * Is this critical or just annoying? The goal of incident assessment is to **evaluate the scope and impact** of a problem, determine its urgency, and trigger the appropriate response. Done right, this step aligns engineering, support, and leadership around a shared understanding of what matters—and what to do next. Let’s break down what effective assessment looks like and how to build your own severity classification system. ### **What Is Incident Assessment?** Incident assessment is the process of determining whether an observed issue qualifies as an incident—and if so, how serious it is. To assess an incident, you typically ask: * What’s broken? * Who is impacted? * Is there a workaround? * How fast do we need to act? The outcome of this process is a **severity level** that maps to your internal response playbook: who gets paged, how quickly you communicate, and how visible the incident becomes across the company. ### **Why Severity Levels Matter** Clear severity definitions help your team: * Act faster under pressure * Escalate the right issues * Prevent over-alerting or under-reacting * Set communication expectations internally and externally They also create psychological safety. When engineers know exactly what qualifies as a SEV1, they don’t waste time debating—they act. ## Severity Levels: Example Framework Here’s a simple, 3-tier severity model you can adopt or adapt: | **Severity** | **Impact** | **Example Incident** | **Expected Action** | | ------------ | ----------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------- | | **SEV1** | Critical / Total Outage | Full production outage, major security breach, data loss | All-hands on deck. Wake people up. 24/7 response. Execs informed. | | **SEV2** | High / Partial Outage | 10% of users can’t log in, degraded performance, partial failure | Escalate to on-call immediately. Frequent updates. Prioritized fix. | | **SEV3** | Moderate / Minor Bug | Broken styling, slow dashboard load, minor UX issue | Fix during business hours. Log the issue. May not require updates. | ### A Score-Based System for Classifying Severity You can use a weighted scoring system that evaluates incidents across five dimensions. This adds structure and reduces subjective decisions: | **Dimension** | **Low (1 pt)** | **Medium (2 pts)** | **High (3 pts)** | | ------------------- | ------------------------- | ---------------------------------- | ------------------------------------------- | | **User Impact** | \<5% affected | 5–25% affected | >25% or all users affected | | **Functionality** | Cosmetic / minor bug | Partial functionality loss | Core feature broken, no workaround | | **Business Impact** | No SLA/revenue/legal risk | Mild SLA concern or revenue impact | Revenue loss, SLA breach, or legal exposure | | **Urgency** | Can wait for a sprint | Fix in a day or two | Requires immediate attention | | **Workaround** | Easy workaround exists | Workaround is possible but painful | No workaround available | Then, you can map the final score as follows: | **Total Score** | **Severity Level** | | --------------- | ------------------ | | 5–7 | SEV3 (Low) | | 8–11 | SEV2 (Medium) | | 12–15 | SEV1 (High) | ### Example: Users on an unusual browser cannot check out Let’s say our business is a review site with an ecommerce store. Users on Microsoft Edge can’t check out due to an incompatibility with our payment provider implementation. * User Impact: Low (1) — Less than 5% of all our users are on Microsoft Edge * Functionality: High (3) — Users are prevented from a final checkout step, and are unlikely to switch browsers, instead abandoning their cart * Business: High (3) — This will cost revenue * Urgency: Medium (2) — At our estimate, this only requires updates to dependencies, and can be fixed in a day or two * Workaround: Medium (2) - We definitely don’t want to add a ‘please switch browsers’ message to our site → **Score: 12 → SEV1** ## Downloadable Incident Severity Cheat Sheet If you want to adopt or adapt this process, you can make a copy of our own [incident severity cheat sheet](https://docs.google.com/spreadsheets/d/18L2r8u5h8ylRWNbfMZv0Ff-amJ-ySk5ZU1FLxW2G4uY/edit?usp=sharing). ## Creating Your Own Severity Rules Every organization operates differently, and what counts as a critical incident for one team may be a routine alert for another. Here’s how to build a severity scoring system that reflects your team’s priorities, customer expectations, and business context. ### **1. Pick Dimensions That Matter to You** Start by identifying the dimensions of impact that are most relevant to your systems and stakeholders. Common ones include **user impact**, **feature impact**, **business risk**, **urgency**, and **workaround availability**—but you might also include **compliance violations**, **customer tier affected**, or **data integrity** if those are key concerns. The goal is to capture the real-world consequences of an incident in a way that reflects your product and risk model. ### **2. Agree on What “High”, “Medium”, and “Low” Mean** Without clear definitions, severity scoring quickly becomes subjective. What one engineer sees as a “minor issue” might be considered “urgent” by someone in customer success. To avoid this, document clear criteria for each level of impact. For example, define “High User Impact” as “more than 25% of users affected” or “SEV1 Business Impact” as “any outage that causes revenue loss or legal risk.” These definitions become your north star for consistent triage. ### **3. Add Automation Where Possible** Manual severity scoring can slow things down and introduce inconsistencies, especially during high-pressure incidents. Automate as much of the process as you can. A shared Google Sheet or Notion template can help teams select impact levels via dropdowns, with scores and severity levels calculated automatically. For more mature teams, connect this logic directly into your alerting pipeline or incident management tool so severity is auto-assigned on alert creation. ### **4. Train Your Incident Responders** Severity models only work if everyone is aligned on how to use them. Run training sessions or tabletop exercises with your incident response team. Use past incidents to “test” the model—how would they score it now, and does the outcome feel right in hindsight? Over time, this helps calibrate judgment, improves consistency across teams, and creates shared understanding between engineering, support, and leadership. ## Final Words Clear severity rules turn gut-feel decisions into structured, confident responses. With a shared scoring model, your team can triage faster, communicate better, and stay focused when it matters most. # Best Practices for Communicating During Outages and Incidents Source: https://www.checklyhq.com/docs/learn/incidents/incident-communication Lack of communication, or wrong communication, can often be the part of incident response that comes in for the most criticism. While it’s critical to define playbooks for communication, this article will talk in general about the best practices for communicating with users during an incident. During the difficult process of incident response, an engineer’s mind quickly goes to fixing the problem. In this author’s time of application development, I’d gladly work in silence for hours as long as I was making progress on a fix. But of course, if our users don’t know what’s happening, it can magnify the impact of an outage the point of causing churn. As the team is trying to identify the root cause and figure out a fix, they’ll always be asking how to communicate with users and when. Lack of communication, or wrong communication, can often be the part of incident response that comes in for the most criticism. While it’s critical to define playbooks for communication, this article will talk in general about the best practices for communicating with users during an incident. ## How soon to communicate about incidents There’s no single standard for communication time with users, ask different operations engineers, and you’ll get very different answers. To determine the correct time to contact users, consider asking the following general questions: * Do you want to notify users when you’re only investigating a problem, not even sure if it’s a technical failure on your service or a false report? * Are you comfortable with users hearing about a failure that may not have an effect on them? For example, would you inform users of an outage even if it was limited to one geographic region, or would you want to limit who saw that communication to those in the affected region? * In the last 5 incidents that generated feedback from users, how often was your team’s communication a major problem? With the answers to these, you should be able to give general guidance to your team so that the person who’s selected by the on-call rotation can make informed decisions about how to notify users. ## Automate communication when possible During total outages, where some or all users are totally unable to use your service, no amount of delay in notifying affected users will feel acceptable to those users. Users expect that the second your service goes down, the login box on your page will be replaced with a ‘technical difficulties, please stand by’ message. While users always have high expectations, this one does feel reasonable: after all shouldn’t it be trivial to detect that a service is completely down, then list that on a [status page](/docs/communicate/status-pages/overview) automatically? On the other hand, we don’t want to tell users not to log in when the service is working fine. So while it’s best to automate status messages, we require a robust tool to detect these outages. You should, wherever possible, have an automated system that shows the availability of your service. Such an automated communication tool should do the following: * Regularly make requests of your key services * Implement retry logic, checking again if any request fails * Make requests from a broad geographic area, not relying on a single network route or availability zone * Output the results of these checks to a publicly available [status page](/docs/communicate/status-pages/overview) Of course, not every outage can result in automated communication. As an example: if a datastore starts returning gibberish records for all users, the service is likely to look healthy to most automated monitors (we’ll note that a sophisticated Playwright check could identify functional problems like this one, but still). As such, it is critical that even with a high quality detection and status communication system, your on-call team still needs the ability to communicate with users, preferably in the same channel as the status automation. This also adds a fallback if the automated system isn’t working during an outage. ## Trust your on-call team As mentioned elsewhere in our guide to incident response playbooks, having a plan for dealing with outages and other issues is a huge source of psychological safety for an on-call team. However, while a plan is a good thing to have, you must always trust that your team will need to make their own decisions. I’ll draw an example from my own experience: > One evening a sales person got in touch with the on-call team saying that the site was completely down. The sales person was at the client’s offices and had the site fail during a training as they were showing the interface to 100 engineers. Our tech support team member tried loading the same, frequently used, page and also got a 404 status. We spent a few minutes troubleshooting, switching to different accounts and making sure we weren’t dealing with an invalid browser cache. “Should we update the main page?” asked the tech support person. Clearly, if the site was completely unusable, we should make an update, but I asked her to wait. At the time I barely knew why I made that decision, but in retrospect I recall that I’d been waiting for the last few minutes for the page to load on a very slow mobile connection in our building’s basement. Sure enough, the page loaded fine. The office network was having a hardware problem. And the sales person on location? They’d been using a VPN to our network without realizing! In the case above, our team was tempted to notify tens of thousands of users of an outage that not one of them would have experienced. There is no hard and fast algorithmic path to deciding if all users need to be notified or a status page updated, there are always small and hard to define indications of an outage’s severity. ## Delegate communications on the on-call team Any documentation of an incident response process will divide the response into separate stages of detection, classification, communication, et cetera. A Resolution Timeline

*The process for a serious outage, with each step done in order and no overlap.* This is a fine mental model of the process, but the reality is less like a layer cake and more like an English trifle. An english trifle, By JIP - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=50966818

An english trifle, By JIP - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=50966818

*As in this classic English desert, the layers of incident response often blend in the real world* If two dozen users get in touch from all over the globe, it’s likely we’ll be communicating with all users before we’ve classified the issue. And often we’ll try to communicate before we’ve found a root cause, meaning communication and investigation are happening in parallel. Finally, engineers would like to deploy a fix before the root cause is understood, blending investigation and repair. It’s to be expected that these tasks in the incident response process would overlap, but if you have one person responsible for both technical and business decisions, this blending means enormous stress. In the anecdote shared in the section above, the support engineer during an incident was responsible for updating the status page, while the on-call ops engineer tried to diagnose the issue, and on-call developers examined recent code changes for possible causes. This kind of delegation is key: a mind that is desperately trying to find the scope and possible cause of an issue has little bandwidth for contemplating a useful status message. Further, entrusting communications to a customer facing team member means that: * The support or sales engineer tasked with communication is likely to understand what exactly this outage means for most users, as users average behavior is more clear to someone who works with users directly. * Customer-facing engineers are often the best people to identify expected behavior for the product, proving enormously helpful during the diagnosis stage. ## Conclusions: Communication Remains a Human Speciality Effective communication during an incident is just as critical as the technical response itself. While engineers naturally prioritize diagnosing and resolving issues, keeping users informed—accurately and promptly—can significantly mitigate frustration and prevent churn. This article has outlined key best practices for incident communication, emphasizing the importance of timing, automation, delegation, and trust in the on-call team’s judgment. Determining *when* to communicate involves balancing transparency with caution—not every investigation warrants a broad alert, but prolonged silence can erode trust. Automating status updates where possible ensures users receive immediate, reliable information during full outages, while still allowing for human intervention in complex or ambiguous scenarios. Most importantly, incident response should not rest on a single individual’s shoulders. Delegating communication to customer-facing team members allows technical staff to focus on resolution while ensuring updates are clear, relevant, and empathetic. Ultimately, incident communication is less about rigid protocols and more about adaptability. Real-world incidents rarely follow a linear process, and teams must be empowered to make judgment calls—whether that means delaying a status update to verify an outage or proactively notifying users before a root cause is found. By combining automation with human expertise, organizations can navigate outages with transparency, efficiency, and minimal disruption to their users. # Incident Management Challenges | Checkly Guide Source: https://www.checklyhq.com/docs/learn/incidents/incident-management-challenges Find out what are the most common incident challenges and how to address them. **System downtime is just the tip of the iceberg.** When something breaks in production—whether it's a full outage, or a degraded feature—it kicks off far more than a technical recovery. Behind every incident lies a flurry of internal coordination, customer communication, and hard questions about what went wrong and why. Internally, teams scramble to assess impact, debug under pressure, and coordinate across tools. Externally, trust is on the line—customers are confused, support tickets spike, and leadership wants answers. That’s why incident management isn’t just a technical process. **It’s a test of your culture, communication, and structure**. Let’s break down the real-world challenges teams face when managing incidents—and how a structured approach can help you respond faster, better, and with more confidence. !\[Incident Management Challenges]\(learn/images/Incident Management Challenges (1).png) ## Early Detection Is Still Hard Despite proactive monitoring tools, teams often struggle to detect incidents early enough. The challenge? Modern systems are deeply interconnected—built on microservices, APIs, cloud infrastructure, third-party components, and continuous deployments. When something starts to break, it rarely announces itself with a dramatic failure. Instead, it’s a single failing endpoint. A slightly elevated latency. A login request that takes 5 seconds longer than usual. At first glance, these symptoms might not seem alarming. But left unchecked, they can cascade into a full-blown outage. [Synthetic monitoring](https://www.checklyhq.com/product/synthetic-monitoring/) and continuous API checks can make a huge difference. However, teams need to agree on what “normal” looks like. Without a shared baseline or alerting logic, it’s too easy to ignore early signs—or drown in noisy alerts that don’t mean anything. In the end, early detection isn’t just about tooling. It’s about tuning, ownership, and continuously improving your signal-to-noise ratio. ## **2. Defining What Is an Incident** *An **incident** is any unplanned disruption or degradation of a service that affects users or business operations and requires a response.* What events categorize as incidents? That’s for your team to decide. Some teams treat any failed check as an incident. Others only classify it as such if customers are impacted or a system is fully down. Without alignment, this leads to chaos. One engineer might escalate a minor error, while another silently fixes a major outage without notifying anyone. Every team defines incidents differently. But without clearly defined severity levels, it’s too easy to either over-alert or under-react. Here’s an example of how you could classify incidents by severity: * **SEV1**: Critical—core features down, customers impacted. * **SEV2**: Partial degradation—users are affected, but workarounds exist. * **SEV3**: Minor bug—non-blocking, but potentially noisy. !\[Incident severity levels]\(learn/images/Incident serverity levels (1).png) The challenge is not just defining these, but aligning everyone—engineering, support, and leadership—on what they mean in practice. ## 3. Coordination and Escalation Can Get Messy When things go wrong, teams scramble. But without a clearly defined incident commander or roles like communication lead or scribe, progress stalls. People either duplicate efforts or wait for someone else to lead. Escalation must be automatic. Everyone should know: when this happens, who gets paged, and who owns the response. ## 4. Postmortems Get Ignored or Misused The post-incident review often turns into a blame game or a checkbox exercise. But a good postmortem is blameless, structured, and actionable. Ask: * What failed—process, tooling, or communication? * What went *well*? * What will we change in the runbook, monitoring, or alert logic? Without real reflection, you’re doomed to repeat the same fire drills. ## 5. Fear Slows Down Response One of the most dangerous challenges in incident management isn’t technical—it’s emotional. When engineers fear being blamed or embarrassed in a postmortem, they become hesitant to speak up. They might delay declaring an incident, hoping it resolves quietly. Or they’ll avoid updating stakeholders out of fear that incomplete information will reflect poorly on them. This slows everything down. Detection is delayed. Communication stalls. Recovery takes longer. The antidote? **Psychological safety.** Teams need to know they won’t be punished for triggering an alert or surfacing a potential issue—even if it turns out to be a false alarm. In a blameless culture: * Engineers feel safe escalating issues early * People focus on improving systems, not assigning blame * Postmortems become honest learning tools, not interrogations If people are afraid of being called out, they'll wait. And in incident response, waiting is the enemy. Your culture determines whether your team moves fast—or freezes. Choose transparency over silence. Choose learning over blame. ## Chaos is the Enemy, Structure is the Fix Incident management is a high-stakes, high-pressure process. But when you build in structure—detection, communication, escalation, resolution, reflection—you replace chaos with clarity. Clear roles, proactive monitoring, and automated status updates can’t prevent every outage—but they ensure your team is never caught flat-footed. # Modern Root Cause Analysis | Checkly Guide Source: https://www.checklyhq.com/docs/learn/incidents/modern-root-cause-analysis Discover how unified metrics, logs, traces, and synthetic tests help you pinpoint failures faster, reduce downtime, and more. Root cause analysis (RCA) in modern distributed systems has evolved beyond manual log digging and static alerts sending emails whenever CPU usage went above 90%. Traditional methods struggle with dynamic, cloud-native environments—where failures cascade across microservices, Kubernetes clusters, and serverless functions. Modern observability transforms RCA by unifying metrics, logs, traces, and events into single data sources, enabling teams to pinpoint issues faster. ### From alerts to root causes: the big leap Not every team will have the depth of Checkly’s synthetic monitoring to watch every common user path and test any critical interaction with a site and its API.

*The Checkly dashboard packs in check frequency, retries, response times, and geographic distribution in one dashboard* Still most teams can rely on some basic monitoring that constantly checks if our services are available, whether a simple pinger or a backend tool like Prometheus black box exporter. That, along with improved tools for users to get in touch with their product teams, means on-call team can rely on knowing about outages within minutes of their start. Why then, is it so hard to find the cause of this outage? Starting with this high level dashboard, sadly many developer’s next stop is here:

*Amazon CloudWatch logs, a great tool but pretty tough if you’re desperate to know what’s broken where without any clues except the time of day* Going from high level dashboards to logs is still common for operations engineers today, but it’s not a great experience for quickly finding root causes. First, without more detail we can’t effectively query and filter logs to see only relevant log lines. Second, modern apps are highly decomposed into microservices. As a result their logs can be stored in multiple places, with multiple conventions, making it exceedingly difficult to connect requests into a clear narrative. Close Up Photos of Sand Grains By Siim Sepp (Sandatlas) - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=22300680

Close Up Photos of Sand Grains By Siim Sepp (Sandatlas) - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=22300680

*To belabor an analogy: imagine you’ve lost your wallet on the beach. You have a tool that tells you what part of the world the beach was in, like a high level dashboard. But logs are like these closeup photos of the sand from various beaches. It’s very hard to find what you’re looking for without a great deal of scrolling, and these closeup photos are exceedingly hard to connect with each other.* As cloud architectures expand into greater complexity, broadening our search for a root cause, we need modern techniques enabled by tools like OpenTelemetry, Jaeger, and Checkly to tackle the problems of modern RCA. ### Key Improvements in Modern RCA: 1. **Automated Correlation** – With distributed tracing and [context propagation](https://www.checklyhq.com/learn/opentelemetry/context-propagation/), we shouldn’t be trying to line up time codes to correlate observability data from multiple services 2. **Context-Aware Alerts** – Alerts prioritize issues impacting business metrics (like checkout errors vs. low-severity logs). Sophisticated routing and status updates on tools like Rootly mean only the right team members get woken up during incidents. 3. **Proactive Detection** – Once solely used for end-to-end testing, synthetic user testing is now available to development and operations teams. Any activity frequently performed by users can be performed in automatic test sessions with deep assertions to make sure the responses look correct. ### Example: A payment service fails intermittently. Traditional RCA might check logs in isolation, wasting hours. With modern observability tools: * **Metrics** show elevated error rates. * **Traces** reveal timeouts between checkout and fraud-detection microservices. * **Logs** highlight a third-party API throttling connections. * **Dashboards** correlate this with a recent deployment, suggesting a rollback. ### Tools for Effective RCA: * **OpenTelemetry** (unified data collection for the back end) * **Prometheus/Grafana** (metrics visualization) * **Jaeger** (distributed tracing) * **Checkly** ([synthetic monitoring](https://www.checklyhq.com/blog/what-is-synthetic-monitoring/) of the frontend, correlated with OpenTelemetry traces) Modern RCA reduces MTTR by replacing guesswork with data-driven decisions. ### Conclusion Modern root cause analysis is no longer about digging through logs. Today’s systems are too complex for old methods. But with the right tools—unified tracing, smart alerts, and synthetic checks—teams can find and fix issues fast. Instead of guessing, you get: * **Connected data** (logs, traces, metrics in one place) * **Clear paths** (see how failures spread across services) * **Faster fixes** (know what broke and why) Tools like OpenTelemetry, Checkly, and Jaeger turn chaos into clarity. The result? Less downtime, happier users, and fewer late-night fire drills. *Want to improve your RCA? Start by connecting your [OpenTelemetry tracing to Checkly](/docs/resolve/traces/overview)—before the next outage hits.* # MTTR Challenges & Solutions | Checkly Source: https://www.checklyhq.com/docs/learn/incidents/mttr-challenges Find out why MTTR targets fail—from weak observability and slow detection to cloud‑native complexity—and apply practical tactics to meet your SLAs. Missing MTTR and the concomitant SLA targets is a common occurrence among large tech teams. We all start each quarter with a goal of always resolving issues quickly and delivering high uptime for our users. So why do we fail on these goals? This article is a list of the common causes of missed MTTR: ### 1. **Observability Tools Are Falling Short** Often the longest period of an incident response is the time spent between knowing you have a problem and understanding what’s causing the problem. This is almost definitional, since once you know the nature of the problem it’s generally possible to fix or at least remediate it pretty quickly, for example with a rollback. But what makes this time to understanding the problem so long? One of the primary challenges is the inadequacy of observability tools. When speaking to enterprise teams, about half of engineers report that their observability solutions require significant manual effort, and that alerts lack sufficient context to triage issues effectively. Imagine you’re trying to solve an outage that’s only affecting some users and some features. You may start with a nice dashboard of front end performance monitoring, but once it’s time to really drill in to root causes, you find yourself paging through AWS CloudWatch logs trying to match the time codes of recent failures to see if any errors were logged. When your goals for mean time to repair are in the minutes, ending up in CloudWatch is setting yourself up for failure. When tools are cumbersome and fail to provide actionable insights, engineers spend more time diagnosing problems, which increases MTTR. ### 2. **Time to Detect (TTD) Is Overlooked** Another factor contributing to high MTTR is the time it takes to detect an issue in the first place. Reducing the mean time to detect ([MTTD](/docs/learn/incidents/reduce-mttd)) is crucial for improving MTTR, as delays in detection can lead to prolonged outages. An entire guide on time to detection is available on our Learn site, but in short the most common failures are: * Reliance on user reporting — if a large class of failures will only ever be reported by users, you’re automatically adding 15 minutes at least to your time to detection. As a rule of thumb, **out of the last 10 incidents, only 1 should be user-reported**. * Low-frequency monitoring — if you’re using an automated system to check your service’s availability, it’s critical to [set the right frequency for those checks](https://www.checklyhq.com/blog/check-frequency/). Checks that are happening only every 15 minutes won’t provide much better time to detection than waiting for users to report problems. * Slow alerting — While most teams doing monitoring think quite a bit about the optimal frequency for those checks, less thought is put into the time spent between detection and sending an alert to the on-call team. On more than one occasion this author has seen teams drastically improve their MTTR just by adopting faster standards for how the on-call team got notified (for example adding alert channels beyond email). Again these concerns and more are in a separate page on [optimizing time to detection](/docs/learn/incidents/reduce-mttd). ### 3. **The Complexity of Cloud-Native Environments** Many organizations are adopting cloud-native architectures, such as containers and microservices, to improve efficiency and scalability. However, this shift has introduced greater complexity, higher data volumes, and increased pressure on engineering teams. Cloud-native architectures have made it harder to discover and troubleshoot incidents. This complexity can significantly delay the time it takes to detect and resolve issues. Why is this issue particularly bad with cloud native architectures? The relative ease of deploying new servers, new stacks, even whole new clusters, means that the complexity of operational architectures has increased exponentially since the advent of cloud native architectures. After the fateful instruction of ‘cattle, not pets’ we have found ourselves swamped with an infinitude of small, disposable containers. That’s great until the herd is sick and we cannot diagnose the cause. If you’re unsure if this is true in your own organization, ask this question of your team: * If I point to any particular microservice, and a select a common user activity, can you tell me with certainty whether taking down that microservice will affect that user activity? I believe that most engineers would honestly answer: “no,” and that this was different before cloud native architecture. ## How Can Companies Improve MTTR? To address these challenges, organizations need to focus on optimizing each stage of the incident response process: ### 1. **Reduce Time to Detect (TTD)** Improving detection times requires robust monitoring and alerting systems that can quickly identify issues. This involves setting low scrape intervals for data collection and ensuring that observability tools can ingest and generate alerts rapidly. Faster detection allows teams to begin remediation efforts sooner, which can significantly reduce MTTR. This is covered in more detail in our page on [optimizing detection time](/docs/learn/incidents/reduce-mttd). ### 2. **Streamline Time to Remediate** Once an issue is detected, the next step is to stop the customer pain by implementing a temporary fix. This can be achieved by: * Providing engineers with actionable data and context in alerts. For example, linking alerts to dashboards that show affected services or customers can help engineers prioritize and resolve issues more efficiently. * Linking alerts with deeper root cause information. While it’s critical to show ‘who is affected and how,’ it’s also supremely important to give some indication of what’s happening internal to our services so that our engineers are not stuck sorting through backend logs trying to match time stamps. For example, Checkly Traces links backend traces with synthetic monitoring data, to show application internals related to any user-facing issue. * Easy rollbacks. Whether it’s deploying rolled back code right from source control with (for example) GitHub actions, or feature flagging to disable a new feature that’s causing trouble, it’s critical that on-call teams are empowered to get the service back to a working state by rolling back the clock. With these tools an on-call team can get the system working for users quickly, followed by a longer period of root cause analysis and postmortem. ### 3. **Time to Remediate over Time to Repair** In our guide on defining MTTR we discussed how the time to remediate (how long it takes to patch the problem such that users are not experiencing an error) versus time to repair (how long it takes to fix the underlying problem). In a simple if outdated example: resetting a server with a memory leak remediates an out-of-memory error, finding and fixing the memory leak repairs the problem. While time to repair is important, it becomes less critical if detection and remediation times are optimized. If customers are no longer affected by the time the underlying issue is being fixed, the impact of a longer repair time is minimized. Critically, teams should have a clear mental model of remediation versus repair, and while they’re investigating the issue they should be asking: ‘do I have enough information to remediate this issue, can I remediate now and look deeper into the problem after users aren’t experiencing errors?’ Teams should still aim to reduce repair times, and clearly mark issues that are ‘remediated but not fixed’ so that technical debt doesn’t overwhelm on-call infrastructure. One consideration is that many incidents don’t have a clear moment when the issue is perfectly fixed. A problem like a memory leak may be possible to resolve 100%, but a request throttling with a 3rd party API may be transient but also hard to permanently resolve. ## Conclusions: MTTR challenges and opportunities It’s very difficult to measure any complex human system. As business conditions change, hard performance measurements will naturally change, but if performance is improving or suffering is harder to determine. The 80/20 principle that most work is wasted effort with a small slice having the highest impact remains true in Operations as in so many technical fields. But if you track your time to resolve downtime incidents and other outages, it’s possible to see if things are getting better or worse. And averaging together performance makes it easier to eliminate too much focus on single incidents and their unique factors. # How to Set an On‑Call Schedule | Checkly Incident Response Source: https://www.checklyhq.com/docs/learn/incidents/on-call-schedule Learn to design effective on‑call rotations—shift models, clear response rules, scheduling tools, mock drills, and fairness to prevent burnout. Once an incident is detected, it’s time to classify the incident and coordinate with your team. To all that, you’ll need an on-call schedule. This guide covers the necessary components for a successful on-call schedule. ## Why Have an On-Call Schedule? Things break. People need help. No matter how well-designed, a software system that is constantly getting new features and scaling to meet demand will experience outages and other incidents. Putatively, an on-call schedule is there to make sure someone is always ready to fix problems, but really they tend to be adopted for another reason: burning out your best. After the first inchoate stage of a software team, there will remain a tendency to bring all technical questions, at any hour of the day, to just 1 or 2 experts. These are often people who are both knowledgeable and possessed of a tendency to look at their phone during dinner. These very effective engineers will quickly get burnt out if they’re the only ones who can actually fix things during a crisis. As such, an on-call schedule can help focus minds as to who can cover during incidents, and who needs more training to get to that level. ## Steps to Set Up Your Schedule First, decide who needs to be on call. Pick people who know how to handle problems and can respond quickly. Make sure everyone gets a turn so no one feels overworked. Next, choose how long each shift will last. Some teams switch every day, while others cover a full week. Think about what works best for your team and how often emergencies happen. Then, set clear rules for when and how the on-call person should respond. Decide what counts as an urgent problem and how fast they need to act. Make sure everyone knows how to reach them, whether by phone, Slack, or an alert tool. After that, use tools to make it easier. A scheduling app like PagerDuty or Opsgenie can send alerts and track who’s on call. A shared calendar helps everyone see the schedule without confusion. Finally, test the schedule before real problems happen. Send a practice alert to make sure it works. Ask the team for feedback and adjust if needed. A good on-call schedule keeps things running smoothly and helps everyone feel prepared. ### 1. Choose Who Is On Call The first and arguably most important step of an on-call schedule is deciding who is eligible to help field reports of emerging issues. The number of available on-call engineers will affect every decision you make afterward. * Pick team members who know how to handle emergencies. * Make sure everyone gets a fair turn. ### 2. Decide the Rotation Next, choose how long each shift will last. Some teams switch every day, while others cover a full week. Think about what works best for your team and how often emergencies happen. You can divide your time by each person on rotation doing a **week and weekend**, where one person covers on-call duties for a full seven days in a row. This gives them the rest of the month free if the team is large enough. While simple to manage, it can be exhausting since the same person handles all alerts day and night for a whole week. Teams with at least four people can rotate monthly, while smaller teams may need to switch every other week. A second option is **workweek and weekend with a backup**, which adds a second person for support in case the primary on-call responder is unavailable. This makes response more reliable but requires at least four team members to avoid burnout. The backup must stay alert, which can be hard to enforce, and managing two schedules adds complexity. If the rate of incidents is low, the backup may go weeks without ever being used, and it’s natural to slack off after a while. The third model, **daylight hours**, spreads coverage across time zones so no one works overnight. Team members only handle alerts during their normal work hours. This protects sleep schedules but requires careful coordination when passing incidents between shifts. Without enough people, some night coverage may still be needed. Larger teams can combine this with longer rotations to give everyone breaks. It’s important to note that your team members will not all feel the same way about these rotations. If possible, let engineers choose a model that makes sense for them, and doesn’t simplistically assign a ‘fair’ system for how easy it is to administer. Remember that a small team can often get by with something like: ‘on-call rotates every day, if you need your day off you’re responsible for finding someone to switch with you.’ ### 3. Set Clear Rules Try to answer questions about on-call expectations early in the process, not during a post-mortem for an incident that went unread for 8 hours. * When should the on-call person respond? (Example: within 30 minutes.) * What counts as an emergency? (Example: system down, major errors.) * How do people reach them? (Example: phone, Slack, PagerDuty.) * What are some activities the on-call person should avoid during their shift? Can they fly on a plane and be offline, is it okay for them to be drunk or use drugs during their on call shift? Answer these questions now, as a team, or face case-by-case problems and embarrassment later! ### 4. Use Tools to Help Unless your team is tiny, you’ll need a few tools to help coordinate on call rotations. * **Rootly, Opsgenie, VictorOps, or something similar** – Automate alerts and rotations, and when more than one person is on-call * **Create a Shared calendar** – Show who’s on call and when. * **Start Simple -** Automated Slack messages might be all you need for your first on-call setup A lot more could be written about how observability tools help during incidents, but those are considered in another guide, read on for more! ### 5. Make It Fair Tech workers are happy to put up with a lot of ambiguity, complexity, and unpredictability in their jobs, but no one likes working in a situation that feels unfair. * Avoid burnout. Don’t put the same person on call too often. - If you’re relying on one person to cover most of the on-call time, it’s time to take staff development seriously, and pair your on-call expert with someone who needs to gain those skills. * Rotate holidays and weekends. - The more geographically diverse your team, the less likely you are to run into schedule conflicts. ### 6. Test It As your team adjusts to a schedule and process, stage mock incidents to test out the process. * Run a practice alert to make sure it works. * Fix any problems before real emergencies happen. Again if you’re having frequent incidents this may not be necessary, but if every incident isn’t following process, it’s time to practice a dry run! ### 7. Keep Improving * **Ask the team for feedback** - It’s critical to perform post-mortems, even after successfully handled incidents, ask ‘what could have gone better’ about the on-call process. * **Continuously update your process** - Ideally your team and your market will be growing throughout each year, so the process that worked last year probably won’t keep working this year. ## Be Kind Being on call can be stressful. Support your team. Say thanks. Help when needed. ### You Have One Team On-call is one of the reasons that people list for leaving teams. Don’t destroy your team while trying to handle incidents, listen to your team members and find a solution that works for you. Spend your energy lifting each other up. A good on-call schedule keeps things running—and people happy. # Incident Response - Improve Recovery & Avoid Downtime Source: https://www.checklyhq.com/docs/learn/incidents/overview Master incident response with real-world insights. Learn how to reduce downtime, regain user trust, and build a resilient incident management workflow. We’ve all been there, the 3AM phone call, the bleary-eye scanning of a Slack channel, the debates over what to say on the status page, the rollbacks, the restarts, and the attempts to find root causes and deploy a fix. Incident management happens every day, and when it’s working well both your users and your leadership may be barely aware of it. But when incidents are severe or when incident management isn’t done well, it’s the only thing anyone wants from your product. In the last decade as the whole world has grown to accept software-as-a-service, the standards for uptime and responsiveness to issues have increased steadily. While day-long maintenance windows and hours-long outages were par for the course in 2015, now even an outage of a few minutes can affect business health. Further, expectations about uptime have gone from ‘best practices’ to ‘binding agreements with financial costs for failures,’ with enterprise clients demanding service level agreements (SLAs) with penalty schedules if uptime goals aren’t met. ## Getting Started Incident Response in the software industry refers to a time-critical response to some kind of availability incident. What makes a good status page? Learn user expectations and technical requirements ## Detect and Resolve Incidents Faster With Playwright ## On the Checkly Blog Learn seven best practices to fight alert fatigue. Strategies to configure your alerts for team success. Five best practices to help you deploy your software more securely and reliably. # Incident Response Playbooks - Culture & Compliance | Checkly Source: https://www.checklyhq.com/docs/learn/incidents/playbooks Master playbooks by setting clear standards, empowering team autonomy, and refining processes when steps are skipped. Incident management is a critical aspect of maintaining system reliability. When engineers are woken up late at night, getting on an incident chat, and trying to fix things while communicating with the team, a huge source of safety and stability is a playbook that describes what to do when. This article serves as a guide for application developers on how to get started writing playbooks for incident response. ## Team Culture in Incident Response Playbooks The term ‘playbook’ sometimes implies a sort of mechanical process that tells engineers exactly what to do, a bit like a robot following lines of code. But of course the term is taken from sport, where a playbook lists individual plays that allow for human participants and implicit randomness. Team culture is a clear part of incident response, and you need to consider how real people will interpret plans when faced with real failures and stressful situations. Two early questions are how much standardization of playbooks should happen between teams, and how to handle cases where people don’t follow the process. ### How much should teams standardize their incident response? A basic question when you start writing playbooks for incident response is how standard these playbooks need to be. Should teams write their own playbooks, or should ops, SRE, or platform teams write the playbook that all other teams need to follow. Standardization across the whole organization can’t be extreme. Let’s say a playbook has a decision point about the scope of an issue. A backend team working in database interfaces and a frontend team will have totally different answers to what ‘scope’ means and how to measure it. So we want standards for how incidents are handled, but now how they’re identified, investigated, or remediated. One senior ops engineer talks about the balance between team standardization and organizational process: > Teams are responsible for their own stuff at the end of the day. They will have their own runbooks as their systems will differ from other teams. The incident response process is standard across all teams, the software teams create and maintain will differ though, so this makes sense as a boundary of responsibility for this organisation. > > Generally teams manage their own alerts and dashboards. If they **continually have problems as a result of poor visibility, it'll be very clear to the leadership due to the number of incident tickets assigned to them**. They'll get the support they need to fix the underlying problem, be that fixing the system or improving the observability. That general truism should apply to your organization: if a team’s individual process for incidents isn’t working, it’ll show up in the number of incidents assigned to the team. ### What to Do When Someone Doesn’t Follow the Process People often skip steps when the process feels slow or hard. If this happens, it means something is wrong—maybe the process is confusing, scary, or takes too long. Others might not ask for help because they had a bad experience before or feel rushed. First, **ask nicely** why they didn’t follow the rules. Don’t blame them—just listen. Maybe they didn’t know how, or the process didn’t work when they needed it. Next, **talk to more people** to see if others feel the same way. Anonymous surveys or quick chats can help find the real problem. Then, **fix the official process** before making the wrong way harder. If the right way is simple and fast, people will use it. Only after that should you add roadblocks to stop the shortcuts. The goal isn’t to punish—it’s to make the best way also the easiest way. ## Conclusions Effective incident response combines automation and teamwork. Automation helps by speeding up tasks and reducing errors, but human judgment is still key. Teams should standardize processes while allowing flexibility for different needs. If people skip steps, the focus should be on fixing the process, not blaming individuals. The goal is to make incident response faster, safer, and easier for everyone involved. By balancing automation with clear, simple processes, teams can handle incidents better and keep systems running smoothly. # Postmortems That Drive Real Improvements (+ Template) | Checkly Guide Source: https://www.checklyhq.com/docs/learn/incidents/postmortems Postmortems aren’t just documentation exercises. Check out how to properly do one + get a downloadable template. The incident is over. The system’s green. Everyone’s relieved. You can go back to sleep now. But without a proper postmortem, you're likely to repeat the same fire drill next month—with the same confusion, stress, and delays. Postmortems aren’t just documentation exercises. Done well, they’re powerful tools for **continuous improvement**, team alignment, and culture-building. In this article, we’ll walk through how to run postmortems that actually make your systems and teams stronger—not just your incident timeline longer. ## Why Postmortems Matter Incidents are inevitable. What separates high-performing teams is not how rarely they fail—but **how well they learn from failure**. A strong postmortem: * Helps your team understand what happened and why * Identifies system or process weaknesses—not people * Surfaces actions to prevent repeat issues * Reinforces good behaviors under pressure Too often, though, postmortems are rushed, overly technical, or just ignored. The result? No follow-through, no learning, no improvement. ### The Firefighter Trap We recently interviewed a senior tech lead who told us about an antipattern called ‘The Firefighter Trap:’ > This is the story of how I fired the Ops Engineer who was consistently the highest rated on the team. Everyone had a story of how the system had gone down in the middle of the night, or how a key user’s data had gotten corrupted with bad database entries, and our favorite Ops Engineer was the one who swooped in and got things working right away. > Unsurprisingly, this engineer became known as the one who put out the most fires, and everyone gave him glowing reviews. The problem was that once we took a look at these incidents, striking similarities showed up: for one set of incidents a race condition could cause a table mismatch. In another, a key service leaked memory badly and needed to be manually restarted. > When I looked at the firefighter’s workload it seemed that all his time went to putting out these fires, and **he wasn’t identifying the underlying issues that caused these outages.** After a short spike to fix these issues, it took two weeks to resolve this tech debt. With a better post-mortem process, we wouldn’t have needed the full-time work of a senior engineer to fix issues manually. It’s great to take a victory lap after an incident is resolved. But you must work to ensure that this problem is handled automatically in the future. Post-mortems, then are a critical step in incident response: Without them you’re likely to find yourself stuck in a loop of responding to incidents without solving their causes. ## What a Great Postmortem Looks Like Here’s the anatomy of a postmortem that actually drives improvement, based on the framework shared in the webinar: ### 1. **Create a Safe, Blameless Space** Psychological safety is the foundation. No one should feel like they’re on trial. Focus on systems, not individuals. Use phrases like: * “What signals did we miss?” * “What could we improve in the process?” * “Where was communication unclear?” If people fear blame, they’ll delay declaring incidents or avoid contributing to future postmortems. Safety enables transparency. ### 2. **Write a Clear, Honest Timeline** Document the incident as it unfolded: * When did the issue start? * When was it detected? * Who responded and what actions were taken? * When was it resolved? Include timestamps, links to check results or traces, and any relevant logs. This isn’t just about storytelling—it’s about understanding time to detect, time to resolve, and where delays happened. ### 3. **Analyze What Went Wrong—and Why** Was there a monitoring gap? A failed alert? A communication bottleneck? A missing runbook? Drill deeper than “the service went down.” For example: * The alert didn’t fire because it was misconfigured * The responder didn’t act immediately because ownership was unclear * The customer wasn't notified because the status page wasn’t linked to the alert channel Go beyond symptoms and look at root causes—technical and procedural. 💡 *What about ‘a third party service went down’? Surely that’s the end of the analysis, right? While you can’t debug another service’s internals, try asking these questions:* * *Next time, can we fail gracefully?* * *Is it possible to use another service as a fallback?* * *Can we monitor this third-party service proactively, e.g., [with an API monitor](https://www.checklyhq.com/learn/monitoring/api-monitoring/)?* ### 4. **Capture What Went Well** It’s easy to focus on failures, but recognizing **what worked** reinforces good behavior and boosts team morale. Did the alert fire correctly? Did someone step up as incident commander? Was customer comms fast and clear? Call it out. Celebrate wins—even in chaos. ### 5. **Define Next Steps With Owners** Turn findings into action: * Add a missing alert or adjust thresholds * Update a runbook * Automate a manual communication step * Clarify on-call roles or escalation paths Each action item should have an **owner** and a **due date**. Otherwise, nothing changes. 💡 *We did a whole webinar on incidents and post-mortems, which you can watch on-demand [here](https://www.checklyhq.com/webinars/incident-management-101-detection-to-communication/).* ## Steal Our Postmortem Template If you want a ready-to-use postmortem template you can steal and/or adapt, use ours: * [Copy the Notion template](https://www.notion.so/Checkly-s-Incident-Postmortem-Template-for-Engineering-Teams-1ecec050b06e805e80b7d714d2a22fb3?pvs=21) * [Download a Google Doc](https://docs.google.com/document/d/1qdLpwkn-qrF5kJk_Y-EFWHmMTiH6jmlRpQ6YFzOlZCU/edit?tab=t.0) ## Postmortems Are Not Just for SEV1s Every SEV1 should have a postmortem. But consider doing them for SEV2s too—especially if: * A monitoring or escalation gap was exposed * Customers were confused due to poor comms * On-call responders were unclear about their role Small incidents are often the warning signs for bigger ones. Don’t miss the learning moment. ## Final Thoughts Postmortems aren't just a checkbox at the end of an incident—they're a lever for building resilience, clarity, and trust. Done right, they strengthen your systems. More importantly, they strengthen your team’s confidence in handling future incidents. # Reduce Mean Time to Detection MTTD - Best Practices Guide Source: https://www.checklyhq.com/docs/learn/incidents/reduce-mttd Avoid user‑reported outages with synthetic checks, anomaly detection, smart alerting, and rich failure traces for rapid detection. When technical organizations enter into Service Level Agreements (SLAs) with clients, along with ensuring reliable code and infrastructure, a significant part of meeting that SLA is ensuring that any downtime events, outages, or other failures are resolved as quickly as possible. Often, teams spend a great deal of time focusing on how problems are investigated, how they’re remediated, and how on-call teams follow playbooks to connect with the relevant engineers to fix the problem. Less focus is given on the first chunk of time spent during any outage: the time between a failure, and anyone in an on-call team knowing a failure has occurred. A Resolution Timeline

Mean Time to Repair is a multi-step process with many areas for improvement, but this article focuses on how to shorten the time it takes to detect a problem. Here are the key optimization areas for mean time to detection. ## Don’t rely on user reports Do you want to break SLA? If you do, rely on your users to report outages. To do some quick math, say you have a ‘three nines’ SLA with your users to provide 99.9% uptime. On average, you have 3 outages a month, and you don’t work on fixing those outages until a user reports a problem. [Use our Check Frequency Calculator to work out your own service's time budget and ideal time to detection.](https://check-frequency-calculator.vercel.app/) 99.9% uptime gives you 43 minutes a month as a ‘time budget’ for all outages. With three outages a month, each outage needs to be resolved in under about 14 minutes. And on average it takes users 15 minutes or more to report an outage to your on-call team. So by relying on user reports of outages, you are guaranteeing that your SLA will be broken on an average month. Notably, many organizations cheat their own metrics by measuring their mean time to repair as starting only after a user reports an outage, when the on-call team is notified. For those teams it seems that their time to detection is always 0 😅. But in reality: user-detected outages add significantly to time to detection and can’t be part of most outages. Steps like adding support hotlines and other faster ways for users to report problems are only papering over the fundamental problem: your tech stack should report its own outages before your users have time to get in touch, every time. How do you stop relying on user reports of problems? In general there are three approaches, all of which should be explored: * Deployment testing — usually the first thing teams try to reduce user-detected problems is to ramp up testing before each code deployment. This boils down to reducing the number of production outages rather than trying to detect them when they occur. I note this route here since it’s such a frequent response, and because the Playwright tool set that we’ll mention later in this piece is also fundamental to thorough pre-deployment end-to-end testing. * Anomaly detection — some teams set up alerts to show when certain backend metrics went over a user-set threshold. One example would be an alert to show when a server was at 99% memory usage. This straightforward approach has its limitations, this author recalls when a network outage blocked most users from our service. We got no alerts because, hey, the servers had plenty of memory, and the wait time on the queue was zero! Anomaly detection goes a step beyond simple alerts and warns when anything ‘looks wrong,’ like a popular ecommerce site that’s seeing zero traffic Sunday afternoon. Anomaly detection, often enabled with an AI model, can provide surprising insights and even warn of some pending failures, but anomaly detection won’t catch every issue, and your users will end up reporting some problems. * Proactive monitoring a.k.a. synthetic monitoring — By actively making requests of your service as a user, the ‘synthetic user’ of ‘synthetic monitoring,’ you can send requests that are a reliable indicator of service available. With requests scripted in Playwright, you can ensure that your synthetic user makes requests just as complex as real users, and that any failure is reported straight to your on-call team with no need to wait for users to pick up the phone. With its power to find outages before your users do, the rest of this article will focus on the right way to user synthetic monitoring to find outages proactively. ## Check Frequently Automated tools like [Checkly](https://www.checklyhq.com/) can make sure that every outage is detected quickly and accurately, but if you set the interval between checks too long, you’ll still be relying on users to report some outages. If no checks of your service are running for more than 15 minutes, it’s highly likely that some outages will be discovered by your users before a synthetic monitor can alert you. For more aggressive SLA’s, you may want to [set your check cadence](https://www.checklyhq.com/blog/check-frequency/) to run every minute from at least one location. ## Mitigate alert fatigue Many of the points listed in this article will revolve around making sure that an alert is emitted as soon as the service isn’t working for some users. But when analyzing slow incident response, there’s another way that our outage detection can fail: alert fatigue. While alert fatigue sounds like a quality of life issue, generating too many alerts can essentially disable your outage detection. If recent incident post-mortems have identified that 1) the recent outage did trigger an alert, but 2) that alert was one of dozens emitted every day, it’s likely that no one is reading those alerts at all. Over-alerting isn’t much better than generating no alerts at all. To control alert fatigue, here are some suggestions in brief: * Take alert fatigue seriously — false positives (alerts when there was no problem) should trigger investigations and fixes just like false negatives (undetected outages). * Set retry policies — requests sometimes fail via the internet. Set a reasonable [retry policy](/docs/communicate/alerts/overview) on your checks so you’re not generating false alarms. * Use smart tests — using the [auto-waiting](https://www.checklyhq.com/learn/playwright/waits-and-timeouts/) and intelligent user-centric assertions of Playwright will help ensure your tests aren’t too strict and therefore flaky (generating lots of false positives). * Route alerts intelligently — use a tool like Rootly or Pagerduty to make sure only the right people are getting alerts. If you take these steps, you can help ensure that a human being is reading all the alerts they receive. The other points included in this article will also help reduce how many alerts your engineers get that they can safely ignore. ## Informative messages The more information you can cram into an alert will drastically improve your time to detection. Part of this is about alert fatigue: an informative alert message means an on-call team member can easily evaluate whether they should take action, and reduces the temptation to ignore all alerts from one channel. Another consideration is how an informative message harnesses our engineers’ capacity to find patterns. An alert about slow response times might be nothing to worry about, but if the alert includes a timeline of response times, the on-call engineer can notice that the slow responses started right when new code was deployed, and that the problem is getting worse! The ability of on-call teams to see the ‘shape’ of an outage from the alert message is critical for proper escalation, sharing of information, and going from an automated message to a human response. If you’re using Playwright, [consider adding steps to your test code](https://www.checklyhq.com/blog/improve-your-playwright-documentation-with-steps/), to improve the documentation of what part of a test failed when notifying the on-call team. ## Simulate complex user paths While other points here have covered the ways we notify the on-call team and what information is offered about the issue. But when we’re faced with false negatives that have missed problems (and forced users to be the first reporters), it’s critical to examine what we’re monitoring and how. In the early days of ‘synthetic user monitoring’ the synthetic user was a headless browser that just checked URL’s for a 200 response. We’ve all had experiences where a service responds with a 200 status code and a body of `Internal service error`. Beyond parsing pages and following links, a true automated browser is necessary to monitor if our services are online. Further we want to have complex [assertions](https://www.checklyhq.com/learn/playwright/assertions/) available to ask the same questions that our users do. For example, when our user visits our ecommerce site, and searches for products, they expect to get more than one result, and they expect each item will have an ‘add to cart’ button. Users care about these things more than they do exact response times or JavaScript errors, so it’s these complex user paths that should be tested when monitoring our service. A tool like [Playwright](/docs/detect/synthetic-monitoring/browser-checks/playwright-support) is the best way to really act like a user when interacting with our service. Checkly uses Playwright to let engineers test every aspect of their service with the same expectations their users bring. ## Gather real evidence of failure A final point, once again related in part to alert fatigue, is making sure that our monitor gathers real evidence of failure. Automated systems can alert for many reasons, and we want to make sure that once a human is investigating it’s easy to differentiate between false alarms and real outages. Three things we’d like to get either at first alert time or shortly thereafter on an alert dashboard. * Scale - A failed synthetic test shows, essentially, ‘One user could not do X with our service.’ The next questions from an on-call team will probably involve scale: how long has this been happening? Can no users do X or only some? Ideally downtime alerts to an on-call team will have at least initial indicators of scale. * Geographic distribution - many failures are specific to one region, and a simple report showing that only some users or affected, or that some requests are successful, will hide the fact that no users in one region have access.

*The Checkly dashboard showing a timeline of failures, that some but not all checks are failing, and listing the geographic locations where checks are failing.* * Traces - How many times have users reported an error, only for the exact error to be difficult to be reproduce, and then the on-call team is trying to replicate a failure with only a vague report to go off of? Ideally every report of a failure would include a full trace showing what loaded successfully, what errors were thrown, and a waterfall chart of page elements’ loading times.

*A Checkly trace showing a full debug session for a failed check.* ## Conclusions: Detect Faster to Fix Faster In this article I’ve argued that detection is a critical step in your time to repair, and thereby critical for meeting your SLA. Detection isn’t just about closer monitoring or a faster rate of monitoring. By engineering notifications and alerts to be informative, timely, and targtetted you can improve time to detection without any increased infrastructure cost, and reduced stress for your on-call team! # Root Cause with Traces - Faster RCA | Checkly Source: https://www.checklyhq.com/docs/learn/incidents/root-cause-with-traces Combine distributed tracing with synthetic checks to pinpoint failures across microservices, preserve request context, and reduce MTTR. Modern applications built with microservices and distributed architectures make root cause analysis (RCA) increasingly complex. When failures occur across dozens of interconnected services, traditional debugging methods fall short. This guide explains how combining distributed tracing with synthetic monitoring creates a powerful approach for faster incident resolution through vastly improved time to understand the root cause of failure. ## The Challenge of Modern Root Cause Analysis One senior operations engineer I spoke to after a webinar mentioned his frustration with root cause analysis in his team: > Every time there's an outage, I go through the same exhausting routine - first I see requests failing, then I scramble to find where it all started, digging through Kiali or whatever logging tool we're using that week until I spot an exception, grab the trace ID, then jump over to Kibana to search for it, only to be met with a mountain of log lines that takes forever to sift through, find another stacktrace pointing to a different microservice, and then repeat the whole frustrating process all over again, hopping between systems and scrolling endlessly until I finally stumble upon the actual root cause, by which point the damage is already done and everyone's breathing down my neck asking why it's taking so long to fix. Consider these common scenarios: * A checkout process fails intermittently, but logs show no errors. * API latency spikes during peak hours with no clear origin. * A frontend component breaks silently after a backend deployment. All of these types of failures could have happened at a SaaS company ten years ago, and while they wouldn’t be fun to troubleshoot at any time, now instead of a code monolith to analyze, we’re likely working with dozens of microservices all interacting in complex ways. Traditional troubleshooting methods struggle because: 1. **Logs are fragmented** across services and systems. 2. **Metrics alone** don't show request flows. 3. **Alerts lack context** and only show the error message received by users, not the scope or suggested causes. While we can use tools like synthetic monitoring to know that something isn’t working correctly, it’s become increasingly difficult to remediate failures, much less find the root cause. ## The Solution: Tracing + Synthetic Checks ### Distributed Tracing Traces provide a complete map of requests as they flow through your system: * Visualize service dependencies * Identify latency bottlenecks * See exact failure points in complex workflows Example trace showing a failed payment process: ``` [Frontend] → [API Gateway] → [Payments Service] → [Fraud Check] → [Database] ↑ 500ms timeout ``` ### Synthetic Monitoring Proactively test critical user journeys with: * Scripted browser checks (e.g., login → add to cart → checkout) * API test sequences with assertions * Geographic performance testing When combined, these approaches: 1. **Detect issues early** before users report them 2. **Preserve context** across frontend and backend 3. **Accelerate RCA** with complete request histories ## Implementation Guide ### 1. Instrument Your Stack * Use synthetic checks for key user flows. * Add OpenTelemetry for tracing. * Correlate trace IDs with check results. ### 2. Build Your Observability Stack | Tool Type | Example Tools | Purpose | | --------- | ------------------------ | ------------------------------ | | Tracing | Jaeger, Tempo, Honeycomb | Visualize request flows | | Synthetic | Checkly, Synthetics | Test user journeys proactively | | Alerting | Rootly, OpsGenie | Route incidents effectively | ### 3. Analyze Incidents When failures occur: 1. Check synthetic monitors for failures 2. Pull traces using the test's trace ID 3. Follow the path to identify: * Which service failed * How long it took to fail * What dependencies were involved ## Real-World Example: E-Commerce Checkout Failure **Scenario**: Customers report payment failures during peak hours **Investigation Steps**: 1. Synthetic check for checkout flow begins failing 2. Trace shows: ``` [Web] → [API] → [Payments] → [Fraud Service] ↑ Timeout after 2s ``` 3. Metrics reveal fraud service CPU saturation 4. Resolution: Scale fraud service pods + add circuit breaker **Result**: MTTR reduced from 47 minutes to 8 minutes ## Key Benefits 1. **Faster Detection** * Catch issues before users do * Get complete failure context immediately 2. **Clearer Diagnostics** * See the full path of broken requests * Stop guessing between frontend vs backend issues 3. **Prevent Recurrence** * Identify weak points in architecture * Build resilience based on real failure patterns ## Getting Started 1. **Start small**: Instrument one critical user journey 2. **Correlate data**: Connect synthetic checks to traces 3. **Expand coverage**: Add more checks as you grow With tracing and synthetic monitoring working together, teams can shift from reactive debugging to proactive reliability engineering. *Ready to implement? See our guide on [Setting Up OpenTelemetry with Checkly](/docs/resolve/traces/import/overview).* # SLA vs SLO vs SLI - What’s the Difference? Comparison with examples Source: https://www.checklyhq.com/docs/learn/incidents/slo-sla-sli Avoid user‑reported outages with synthetic checks, anomaly detection, smart alerting, and rich failure traces for rapid detection. When we talk about keeping services running smoothly, we often hear about SLAs, SLOs, and SLIs. But what do these terms mean, and how are they different? SLAs, or Service Level Agreements, are like promises between a service provider and a customer. They outline what the customer can expect in terms of service quality. SLOs, or Service Level Objectives, are the specific goals that service providers aim to hit to meet the promises made in the SLAs. Think of them as targets for how well the service should work. Finally, SLIs, or Service Level Indicators, are the measurements used to see if the service is hitting its targets. They help us understand how well the service is doing. Together, these three help ensure services are delivered well and customers are happy. | **Categories** | **SLA (Service Level Agreements)** | **SLO (Service Level Objectives)** | **SLI (Service Level Indicators)** | | ---------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | **What is It?** | A contractual commitment defining agreed-upon expectations between a service provider and customers. | Specific, measurable goals set within the broader scope of SLAs. | Specific metrics measuring a service's performance. | | **How Does It Help?** | Outlines metrics, response times, and service quality to ensure performance standards. | Pinpoints desired performance levels, emphasizing reliability and user satisfaction. | Measures specific aspects of a service's performance to assess its quality. | | **Who Builds It?** | A collaborative effort involving service providers and customers, often led by technical teams. | A collaborative effort involving technical teams to set measurable goals within the SLA framework. | Developed by technical teams to measure and monitor specific aspects of service performance. | | **What Happens if It's Breached?** | Breaching SLA terms may lead to penalties, legal consequences, and reputation damage to the provider. | Breaching SLOs signifies a failure to meet performance goals, triggering corrective actions and potential reevaluation. | SLI breaches indicate deviations in specific performance metrics, requiring investigation and improvement. | ## What is an SLA (Service Level Agreement) At its core, a [Service Level Agreement (SLA)](https://www.techtarget.com/searchitchannel/definition/service-level-agreement) defines the expectations between service providers and customers. Overcoming challenges such as technical intricacies, customer preferences, language clarity, and detailed documentation is key to optimizing SLA effectiveness. By embracing best practices, SLAs become dynamic tools fostering transparency, accountability, and customer satisfaction. ### SLA Challenges Hitting your SLAs can bring many challenges that demand a nuanced and strategic approach. Understanding and addressing these challenges is important for the success and effectiveness of SLAs: * **Defining Precise Metrics:** Accurately quantifying key performance indicators is a fundamental challenge in defining SLAs. The process requires clear definition and measurement to align with both client expectations and operational capabilities. * **Balancing Flexibility and Specificity:** Achieving the right balance between flexibility and specificity is crucial. Overly rigid SLAs may hinder innovation, while overly lenient ones can lead to unmet expectations. Striking the balance is imperative for long-term success. * **Adapting to Evolving Technologies:** The dynamic nature of industries and constant technological advancements pose a continuous challenge. SLAs must be flexible enough to adapt swiftly to changes, ensuring they remain relevant and effective in evolving business landscapes. * **Effective Communication and Collaboration:** Successful SLAs hinge on effective communication and collaboration between service providers and clients. Clear understanding, transparent dialogue, and collaborative problem-solving are essential to preempt and address potential issues. * **Monitoring Mechanisms:** Implementing robust mechanisms for [monitoring service level agreements](https://www.checklyhq.com/blog/how-to-monitor-all-the-nines-of-your-service-level-agreements/) is crucial. Regular assessments and timely feedback loops help identify and address deviations, ensuring service levels meet agreed-upon standards consistently. * **Commitment to Continuous Improvement:** SLAs are not static documents. They are living agreements that demand a commitment to continuous improvement. A proactive approach to refining processes and adapting to changing circumstances is necessary for sustained success. ### SLA Best Practices To overcome these challenges and ensure the effectiveness of SLAs, certain best practices should be observed: * **Involve Technical Teams in SLA Creation:** Collaboration with technical teams from the initial stages ensures that the SLA aligns with the technical capabilities and limitations of the services. This collaboration fosters a more accurate and realistic set of expectations. * **Create SLAs Keeping Customers’ Preferences in Mind:** Considering customers’ preferences is crucial. By incorporating customer feedback and expectations, SLAs become more customer-centric. This also leads to increased satisfaction and trust. * **Keep SLAs Simple and Use Clear Language:** Maintaining simplicity in SLAs is a best practice that cannot be overstated. Clear and straightforward language enhances understanding and reduces the risk of misinterpretation. * **Document Everything:** Comprehensive documentation is crucial for successful SLAs. Documenting all aspects of the agreement ensures transparency. It also provides a reference point for dispute resolution and aids in continuous improvement. ### Who Needs an SLA? Understanding who benefits from SLAs is essential for businesses looking to establish effective service standards. In essence, SLAs are beneficial for: * **Service Providers:** They set clear expectations and define performance standards. * **Customers:** They have a transparent understanding of the services they can expect. * **Businesses:** SLAs contribute to accountability, transparency, and customer satisfaction, ultimately impacting the bottom line positively. ### SLA Examples To illustrate the practical application and importance of effective SLA management, let's explore some real-world examples across various industries: | **Use Cases** | **Description** | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Cloud Services | This SLA between a cloud service provider like Checkly and its customers specifies uptime guarantees (e.g., 99.9% uptime), data security standards, and disaster recovery protocols. | | IT Support | An agreement detailing response times for IT support requests, resolution times based on the severity of issues, and the modes of support available (e.g., phone, email, chat). | | Telecommunications | SLA for a telecom company can include network availability targets, call quality standards, and maintenance window notifications. | ## What is an SLO (Service Level Objective) [Service Level Objectives (SLOs)](https://www.gartner.com/en/information-technology/glossary/slo-service-level-objective) are critical to managing and maintaining reliable and efficient systems. An SLO is a set of quantitative measures that define the level of service a system must deliver. This helps teams align their performance goals with user expectations. SLOs play a pivotal role in ensuring that services meet user needs while allowing organizations to manage their resources effectively. ### SLO Challenges Implementing SLOs comes with a set of challenges. Teams often face difficulties in defining precise and meaningful objectives and striking the right balance between aggressiveness and achievability. The challenge lies in creating objectives that align with user expectations while being realistic regarding system capabilities. Additionally, unforeseen contingencies can impact the achievement of SLOs, requiring continuous adaptation and improvement. ### SLO Best Practices To overcome challenges associated with SLOs, it is essential to follow best practices that streamline the process and enhance the effectiveness of these objectives: * **Keep SLOs Simple and Clear:** Simplicity is key when defining SLOs. Clear and straightforward objectives facilitate better understanding and communication across teams. Avoid overly complex or ambiguous metrics that can lead to confusion and misalignment. * **Account for Contingent Issues:** Recognizing that unforeseen issues can impact service levels is crucial. Build flexibility into your SLOs to account for contingent issues. This allows teams to adapt and maintain service quality despite unexpected challenges. * **Create SLOs for Internal Systems:** While SLOs are often associated with customer-facing services, internal systems also benefit from performance metrics. Implementing SLOs for internal services ensures that the entire infrastructure operates at optimal levels. This contributes to overall organizational efficiency. * **Don’t Create More Than Necessary SLOs:** Creating an excessive number of SLOs can be counterproductive. Focus on the most critical aspects of your services and establish a manageable set of objectives. This enables teams to prioritize effectively and dedicate resources where they are most needed. ### Who Needs an SLO? The adoption of SLOs is not limited to specific roles or teams. Anyone involved in delivering, managing, or maintaining services can benefit from implementing SLOs. Development teams, operations teams, and leadership play crucial roles in defining and achieving SLOs. SLOs serve as a unifying metric that aligns the efforts of various teams toward a common goal—ensuring a high-quality user experience. ### SLO Examples To demonstrate how Service Level Objectives (SLOs) set the stage for measuring and achieving service quality, here are examples from various industries. | **Use Cases** | **Description** | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | E-commerce Website | An SLO for an e-commerce platform may include a page load time of under 2 seconds for 95% of all page views to enhance user experience and reduce bounce rates. | | Online Banking | For an online banking service, an SLO can specify a transaction success rate of 99.5%, ensuring reliability and trust in digital transactions. | | Cloud Storage | A cloud storage service can have an SLO that guarantees data retrieval times of less than 300 milliseconds for 99% of requests, providing quick access to stored information. | ## What is an SLI (Service Level Indicator) [Service Level Indicators (SLIs)](https://www.techtarget.com/searchcustomerexperience/definition/service-level-indicator) are fundamental components of service level management. They provide measurable metrics to evaluate the performance of a system. SLIs are specific, quantifiable measurements that give insights into various aspects of a service. This enables teams to assess the service’s reliability and effectiveness. ### SLI Challenges Implementing SLIs comes with its share of challenges. Defining metrics that accurately represent the user experience can be complex. Teams often struggle with selecting the right indicators that align with user expectations and business goals. Additionally, ensuring that SLIs remain relevant and meaningful over time requires continuous attention and adaptation. ### SLI Best Practices Overcoming challenges associated with SLIs involves following best practices that enhance their accuracy and relevance. * **Create Precise and Measurable SLIs:** SLIs should be crafted with precision, reflecting the specific aspects of a service that matter most to users. Measurable metrics allow for objective evaluation and facilitate data-driven decision-making. Avoid vague or overly broad indicators to ensure the effectiveness of SLIs. * **Keep SLIs Simple:** Simplicity is key when designing SLIs. Clear and straightforward indicators are easier to understand and communicate across teams. Avoid unnecessary complexity that can lead to confusion and misinterpretation of performance metrics. ### Who Needs an SLI? The importance of SLIs extends to various roles within an organization. Anyone involved in the development, deployment, or maintenance of services can benefit from incorporating SLIs into their processes. Development teams use SLIs to monitor the impact of code changes. Operations teams leverage SLIs to ensure system reliability. Leadership relies on SLIs to make informed decisions about resource allocation and strategy. ### SLI Examples To refine our understanding of service measurement further, let's examine some Service Level Indicators (SLIs) that quantify the performance of services. | **Use Cases** | **Description** | | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Website Uptime | For a web hosting service, an SLI can measure the percentage of time the hosted websites are accessible to users, aiming for an uptime of 99.9%. | | API Response | In API services, an SLI could be the average response time for API calls, with a target of responding within 500 milliseconds for 95% of the requests. | | Customer Support Response | For a customer support team, an SLI can track the average response time to customer inquiries, with a goal of responding to 90% of inquiries within 1 hour. | ## Why are SLAs, SLOs, and SLIs Important? Service Level Agreements (SLAs), Service Level Objectives (SLOs), and Service Level Indicators (SLIs) are integral components of effective service management. Each of them serves a unique purpose in ensuring the delivery of high-quality services. Understanding their importance is paramount for organizations striving to meet user expectations and maintain operational excellence. ### Ensuring Accountability with SLAs SLAs set the foundation for accountability and transparency. These agreements define the expected level of service a customer can anticipate. It also outlines measurable metrics such as response times, uptime, and resolution periods. By clearly defining these expectations, SLAs foster trust between service providers and customers. When complied with, SLAs help organizations demonstrate their commitment to delivering reliable and timely services. ### Aligning Objectives with SLOs SLOs bridge the gap between user expectations and system capabilities. These objectives establish quantifiable performance goals. This allows teams to align their efforts with user needs. SLOs serve as a roadmap for maintaining service quality. They help organizations strike a balance between ambitious targets and achievable benchmarks. Establishing SLOs encourages continuous improvement, adaptability, and a proactive approach to managing service levels. ### Gaining Insights Through SLIs SLIs provide a granular view of service performance. These indicators offer specific, measurable metrics that serve as the building blocks for SLOs. SLIs enable teams to monitor various aspects of service. These range from latency and error rates to user interactions. By regularly evaluating SLIs, organizations gain valuable insights into the real-time health of their services. This process empowers them to make informed decisions, identify areas for improvement, and respond promptly to emerging issues. ### The Collective Impact When integrated, SLAs, SLOs, and SLIs form a comprehensive framework for service excellence. SLAs provide a contractual foundation, SLOs establish performance objectives, and SLIs offer tangible metrics to measure success. This triad ensures a holistic approach to service management, aligning customer expectations with organizational capabilities. ## Practical Example: Implementing SLIs, SLOs, and SLAs for APIs In the context of managing a customer-facing, business-critical API, it's essential to establish clear standards and expectations to ensure high-quality service. Here, we dive into an example that outlines SLIs, SLOs, and SLAs, using a practical scenario. ### SLI: Service Level Indicator The SLI acts as a measure of the performance and reliability of the API. In this case, the SLI is defined by the API's ability to respond successfully with HTTP status codes ranging from 200 to 499, coupled with a response time of under one second. This metric is crucial because it quantifies the operational performance of the API from a technical standpoint, focusing on availability and speed. ### SLO: Service Level Objective Building on the SLI, the SLO sets a target for the level of service the API aims to deliver. For our API, the objective is that the SLI conditions (response codes within 200-499 and response times below one second) are met for the 99th percentile of all requests. This means that out of 100 requests, at least 99 should satisfy these criteria. The SLO is a commitment to maintaining a high standard of service, ensuring that nearly all requests are handled efficiently and effectively. ### SLA: Service Level Agreement The SLA transforms the SLO into a formal agreement with the customer. It guarantees that the service will achieve the SLO targets for a specified period, in this case, a quarter. The SLA also outlines the compensation customers will receive if the service fails to meet these expectations. This compensation could take various forms, such as financial credits, discounts, or other remedies. The SLA is a crucial part of customer contracts, providing a legal framework that ensures accountability and offers reassurance to customers about the reliability of the service. By setting these SLIs, SLOs, and SLAs, the company not only commits to delivering a high-quality API service but also provides transparency and trust to its customers. This framework helps manage expectations, foster customer satisfaction, and drive continuous improvement in service performance. ## **How Checkly Can Help You Hit Your SLAs** [Checkly](https://www.checklyhq.com/) focuses on Synthetic Monitoring to track how well websites, applications, and APIs work. Its goal is to help [meet Service Level Agreements (SLAs)](https://www.checklyhq.com/blog/how-to-monitor-all-the-nines-of-your-service-level-agreements/) with customers with its features such as API checks, browser checks, heartbeat monitoring, etc. ### **API Checks** [API checks](https://www.checklyhq.com/blog/what-is-api-monitoring/) monitor vital API endpoints frequently from various global locations. They allow for the validation of response codes and bodies to ensure accuracy, while also keeping an eye on response times to maintain a quick and efficient experience. Additionally, the feature of receiving instant alerts when any monitoring checks indicate a failure provides the assurance needed for maintaining smooth API operations. This proactive approach to monitoring ensures that APIs function seamlessly, enhancing reliability and user satisfaction. a checkly dashboard

*Check out [this tutorial](https://www.youtube.com/watch?v=38ZXJy-nlvI) to find out how Checkly’s API checks work.* Checkly’s API checks can help you achieve your SLAs through: * **Continuous Monitoring:** Checkly allows you to continuously monitor your APIs from multiple global locations. This helps ensure that your services are available and responsive across different regions, aligning with SLA requirements for uptime and performance. * **Automated Testing in Production:** You can automate API testing to validate the functionality, performance, and reliability of your endpoints. This includes checking for correct status codes, response times, and validating response bodies against expected outcomes. API monitoring helps in the early detection of issues that could breach SLA terms. * **Alerting and Notifications:** Checkly provides real-time alerts and notifications when your APIs do not meet predefined thresholds or when failures occur. This immediate feedback loop enables you to quickly respond to and resolve issues before they impact your SLA commitments. * **Customizable Check Intervals:** You can customize how frequently your APIs are checked, allowing for more frequent monitoring of critical services that have strict SLA requirements. This ensures that any downtime or performance degradation is detected and addressed promptly. * **Performance Tracking:** Checkly tracks the performance of your APIs over time, offering insights into trends and potential areas of improvement. This data can help you optimize your services to not only meet but exceed SLA expectations regarding response times and reliability. * **Detailed Reporting:** The platform provides detailed reports and dashboards that give visibility into API health, performance metrics, and historical data. These insights can be used to demonstrate compliance with SLAs during audits and reviews with stakeholders. ### **Browser Checks** On the other hand, Checkly’s Playwright-based [browser checks](/docs/detect/synthetic-monitoring/browser-checks/overview) mimic user actions to make sure key processes work smoothly, and the heartbeat feature checks if systems are running properly. These capabilities enable the monitoring of response times, uptime, functionality, and internal systems performance. This comprehensive monitoring supports SLAs in keeping services healthy and running efficiently. a checkly dashboard

Checkly’s browser checks can help you achieve your SLAs through: * **Real User Simulation:** Checkly's browser checks use real browsers to simulate user actions, such as clicking links, filling out forms, and navigating through web pages. This allows you to test and monitor the end-to-end user experience, ensuring that your application meets SLA requirements for functionality and user satisfaction. * **Global Coverage:** By running browser checks from multiple locations worldwide, you can ensure that your web application delivers a consistent user experience across different geographical regions. This is particularly important for SLAs that specify performance standards across diverse user bases. * **Performance Metrics:** Checkly provides detailed performance metrics, such as page load times, which are crucial for meeting SLAs related to website speed and responsiveness. Monitoring these metrics allows you to identify and address performance bottlenecks before they affect user satisfaction. * **Visual Regression Testing:** You can use Checkly to perform [visual regression testing](https://www.checklyhq.com/blog/visual-regression-testing-with-playwright/), which ensures that your web application's visual elements render correctly across different browsers and devices. This helps maintain a high-quality user interface, in line with SLA standards for usability and design. * **Error Detection and Alerting:** Checkly alerts you in real-time if a browser check fails, enabling you to quickly identify and resolve issues such as broken links, malfunctioning features, or downtime. This rapid response capability is essential for adhering to SLAs that stipulate minimal downtime and quick issue resolution. * **Customizable Check Intervals:** You can configure the frequency of your browser checks to match the criticality of different application components. For example, you might run checks on key user flows every few minutes to ensure high availability and performance, aligning with stringent SLA requirements. * **Reporting and Insights:** Checkly provides comprehensive reports and dashboards that offer insights into the historical performance and reliability of your web application. These insights can be used to demonstrate compliance with SLAs during stakeholder reviews and to identify areas for improvement. *To find out more about Checkly’s browser checks & how to get started, check out [this article](/docs/detect/synthetic-monitoring/browser-checks/overview).* ### **Checkly Best Practices** If you’re using Checkly, here are some best practices that will help you make sure you’re doing everything you can to adhere to your SLAs. * [Monitor your critical APIs more frequently](https://www.checklyhq.com/blog/how-to-monitor-all-the-nines-of-your-service-level-agreements/). To ensure that you’re always aware of the health of your critical APIs, ping them in an interval between 10s and 2min. * Use [parallel scheduling](https://www.checklyhq.com/blog/parallel-scheduling/) to detect regional outages as quickly as possible and [reduce your MTTR](https://www.checklyhq.com/blog/what-is-mean-time-to-repair-mttr/). * Use [smart retries](https://www.checklyhq.com/docs/communicate/alerts/configuration#retry-strategies)–pick one of the three retry strategies we offer based on the frequency of your check runs. *The impact of check frequency on your SLAs:* a checkly dashboard

### **Integrating Into Your Existing Workflow** Checkly enables you to monitor your SLAs by allowing you to monitor services closely and check how they perform worldwide in over 20 locations. You get instant alerts when there's any issue, helping you react quickly to fix it. The platform keeps an eye on services all the time and adjusts to new needs or changes. Checkly integrates on-call tools like PagerDuty and Opsgenie for handling problems, and you can also set up your own connections with webhooks. This helps fix issues fast and keeps things running smoothly. Moreover, Checkly can integrate with your continuous integration and deployment (CI/CD) pipelines, allowing for automated checks to run as part of your development process. This ensures that any changes to your services maintain or improve compliance with SLA requirements before deployment to production. Checkly makes it easier for organizations to set up their monitoring in a way that meets their needs, looks after their services everywhere, and keeps their standards high by quickly dealing with any problems that come up. ## **Conclusion** To put it simply, knowing what SLA, SLO, and SLI mean is really important for anyone working with services, whether you're providing the service, working within a team, or you're the customer. * **SLIs** are like measuring how well your service is doing, similar to checking your health. * **SLOs** are like your health goals, giving you something to aim for in how well your service should work. * **SLAs** make everything official, giving both the service provider and the customer clear rules and protection under the law for what the service should be like. Think of these three terms as the basic parts you need for managing a service that's responsible, high-quality, and always getting better. Whether you use them on their own or together, they help make sure you're offering a great service and always looking to do better. Checkly can become your most valuable partner in achieving your SLIs, SLOs, and SLAs. [Schedule a customized demo to find out how.](https://calendly.com/d/5gk-49g-f76/checkly-demo?month=2024-02) # What is Incident Response? Fundamentals and best practices Source: https://www.checklyhq.com/docs/learn/incidents/what-is-incident-response Learn the key types and stages of incident response—outages, performance issues, security events, user‑reported problems—and keep your services resilient. "Incident Response" in the software industry refers to a time-critical response to some kind of availability incident. The terms are a bit vague since, although many incidents are also outages, not every incident is an outage of a service where it's unavailable to users. Whenever we need to gather one or more engineers to work outside of their scheduled work hours to handle a problem with our service, it's an incident. This intro page lists common incident categories, and the stages of incident response. ### Types of Incidents Understanding the different types of incidents is key to managing them well. Here are the main types: 1. **Service Outages**: These are big problems where a key service stops working. For example, if the power goes out. These need quick action and teamwork to fix. 2. **Performance Degradation**: This happens when a service is still working but not as well as it should. For instance, if a website loads very slowly. It’s important to find and fix the cause fast to keep the service good. 3. **Security Incidents**: These involve threats to the safety of a company’s IT systems, like a data leak. They need special care to protect important information. 4. **User Issues**: These are problems reported by users which, while they may not be caused by a systemic failure on your service’s part, are still affecting multiple users. For example, internet weather affecting a large region, or common user confusion with a new feature or account regulation. A final example would be multiple users who have been affected by malware, which is in turn affecting your service. By knowing these types, teams can handle each one in the best way. This helps keep services running smoothly and keeps customers happy. ### Stages of Incident Response The incident response process helps teams handle problems quickly and effectively. As noted in the list, some stages may not happen in this order; for example the incident may be closed and reviewed with a root cause resolution not being deployed until later. Here are the key stages: 1. **[Identify the Incident](/docs/learn/incidents/detection-best-practices/)**: The first step is to notice the problem. This could be a user reporting an issue or an alert from a monitoring system. The team needs a clear way to know when something is wrong. 2. **Classify the Incident and Determine its Scope**: Once the issue is identified, it is recorded in a system. The team then decides how serious it is and who should handle it. On small teams this will generally be a choice between a level 2 incident handled by the on-call engineer and Support/Sales to handle customer communication; and a level 1 incident handled by pretty much everybody. 3. **Communicate with Users**: Whether it’s through a manual update process, a social media update, or an automated status page, once the incident has been classified and the scope determined it’s time to talk to your users (some teams may choose to be more targeted if the failure only affects a small subset of users). 4. **Remediate the Failure**: In some cases it may be possible to remediate the problem before fully diagnosing the cause. For example, a user-reported bug that is easy to reproduce, that starts being reported one hour after a major code deployment, can likely be remediated with a rollback. This can be done before understanding exactly why the recent deployment caused this error. 5. **Diagnose the Incident**: The team investigates to find the root cause of the problem. At this 6. **Repair the Issue**: Once the root cause is known, the team fixes the issue. This could mean adding more resources, fixing a network problem, or applying a permanent solution. 7. **Close and Review the Incident**: After the problem is solved, the team closes the incident and reviews what happened. This helps them learn from the issue and improve for the future. A Resolution Timeline

By following these stages, teams can handle incidents efficiently, reduce downtime, and keep services running smoothly. ## Best Practices of incident response Each of the stages of incident response has associated best practices: 1. **Don’t let your users be the ones reporting incidents**: whether it’s through monitoring with OpenTelemetry or automated checks with a synthetics tool like Checkly, you don’t want to have any type of failures be reported by users as a matter of course. 2. **Classify the Incident and Determine its Scope**: Once the issue is identified, it is recorded in a system. The team then decides how serious it is and who should handle it. On small teams this will generally be a choice between a level 2 incident handled by the on-call engineer and Support/Sales to handle customer communication; and a level 1 incident handled by pretty much everybody. 3. **Communicate with Users**: Whether it’s through a manual update process, a social media update, or an automated status page, once the incident has been classified and the scope determined it’s time to talk to your users (some teams may choose to be more targeted if the failure only affects a small subset of users). 4. **Remediate the Failure**: In some cases it may be possible to remediate the problem before fully diagnosing the cause. For example, a user-reported bug that is easy to reproduce, that starts being reported one hour after a major code deployment, can likely be remediated with a rollback. This can be done before understanding exactly why the recent deployment caused this error. 5. **Diagnose the Incident**: The team investigates to find the root cause of the problem. Depending on how well our remediation steps worked, root cause analysis may wait until after the incident, or be held off until working hours the next morning. However, if effective remediation isn't available (for example if a simple restart hasn't fixed things), you may be trying to diagnose root causes under time pressure. Key tools include [Checkly Traces](/docs/resolve/traces/overview) to connect [OpenTelemetry traces](https://www.checklyhq.com/learn/opentelemetry/traces/) with synthetics data. 6. **Repair the Issue**: Once the root cause is known, the team fixes the issue. This could mean adding more resources, fixing a network problem, or applying a permanent solution. 7. **Close and Review the Incident**: After the problem is solved, the team closes the incident and reviews what happened. This helps them learn from the issue and improve for the future. Explore the documentation linked above to go deeper with your incident response skills! # Centralized logging in Kubernetes with OpenTelemetry - guide with examples Source: https://www.checklyhq.com/docs/learn/kubernetes/centralized-logging Monitoring logs effectively in a Kubernetes environment requires tools that centralize and streamline log management. Monitoring logs effectively in a Kubernetes environment requires tools that centralize and streamline log management. With OpenTelemetry, Kubernetes administrators can collect, process, and export logs from applications and infrastructure, ensuring consistent data and reducing the need for multiple, disparate logging solutions. Here’s an overview of how OpenTelemetry supports centralized logging in Kubernetes and the practical benefits it offers. ### Why Centralized Logging Matters in Kubernetes Logs provide critical insights into application performance, errors, and security events. In a Kubernetes setup, workloads are distributed across many nodes, making it essential to have a centralized logging solution to collect logs in one place. Centralized logging enables teams to efficiently manage logs from across the entire Kubernetes cluster and prevents loss of logs during pod restarts or node replacements. The OpenTelemetry project, initially created to address distributed tracing, has expanded to include metrics and now, comprehensive logging support. With its recent logging capabilities, OpenTelemetry provides a unified framework for collecting and correlating logs alongside traces and metrics, ensuring consistency and reliability. ### Key Components of OpenTelemetry for Centralized Logging 1. **OpenTelemetry Collector**: The [OpenTelemetry Collector](/docs/learn/opentelemetry/what-is-the-otel-collector/) is at the heart of centralized logging with OpenTelemetry. It’s a standalone agent that can be deployed on Kubernetes nodes, Windows, or Linux hosts, and it’s capable of collecting logs from multiple sources. In Kubernetes, the collector can capture logs, system metrics (such as CPU and memory usage), and metrics from third-party applications. 2. **Instrumentation Libraries**: OpenTelemetry includes libraries and SDKs for popular programming languages, which can add logging capabilities directly to application code. By using OpenTelemetry’s appenders, developers can export logs in OpenTelemetry’s native format, simplifying log management and reducing the overhead typically required for parsing and reshaping logs. 3. **Data Model and Semantic Conventions**: OpenTelemetry introduces a standardized data model and semantic conventions, ensuring that logs have consistent metadata and structure across the entire system. This is critical for efficient querying and analysis, as it eliminates the need for complex parsing and filtering. ### Benefits of Using OpenTelemetry for Centralized Logging OpenTelemetry’s approach to centralized logging addresses some common challenges in log management: * **Consistency in Metadata**: One of the biggest issues with traditional logging is inconsistent metadata. With OpenTelemetry’s semantic conventions, metadata is standardized across logs, traces, and metrics, making it easier to analyze data across multiple sources. * **Performance Optimization**: By processing logs directly in OpenTelemetry’s format, the collector minimizes CPU and memory usage associated with parsing and reshaping logs. Logs from OpenTelemetry-enabled applications can be sent directly to the collector without intermediary processing steps, saving resources. * **Correlation Across Data Types**: OpenTelemetry simplifies correlating logs, traces, and metrics. This correlation provides a holistic view of application performance, helping teams diagnose and resolve issues quickly. For instance, logs from a failed request can be directly linked to corresponding trace data, helping teams identify the root cause faster. ### How the OpenTelemetry Collector Works in Kubernetes The OpenTelemetry Collector can be deployed as a DaemonSet in a Kubernetes cluster, running as an agent on each node. This setup allows the collector to: * Capture logs from containers, application components, and system processes. * Process and transform logs to match the OpenTelemetry data model. * Export logs to different backends (e.g., Elasticsearch, Splunk, or a cloud provider’s logging service). The collector includes built-in parsers and transformation capabilities to handle logs in various formats, such as JSON or text. This flexibility is especially useful in Kubernetes environments where logs from different applications and services may not share a standard format. For more detail on deployment options for the collector, [see our guide to collector architecture](/docs/learn/opentelemetry/what-is-the-otel-collector/). ### Implementing Centralized Logging with OpenTelemetry in Kubernetes Here’s how to set up centralized logging in Kubernetes with OpenTelemetry: 1. **Deploy the OpenTelemetry Collector**: Deploy the OpenTelemetry Collector as an agent on each Kubernetes node. This enables it to capture logs from all pods running on that node. A DaemonSet can simplify deployment, ensuring the collector runs on every node in the cluster. 2. **Configure Log Ingestion**: Configure the collector to ingest logs from different sources. The collector supports various log sources, including container logs, syslogs, and even other log agents like Fluent Bit. It can read application logs directly from files or through network protocols. 3. **Define Transformations and Exports**: Use OpenTelemetry’s processing capabilities to normalize logs, add metadata, and apply semantic conventions. After processing, configure the collector to export logs to a centralized logging backend or a cloud-based logging service. 4. **Correlate Logs with Traces and Metrics**: The collector can correlate logs, traces, and metrics by applying consistent metadata and sampling rules. For instance, if you’re sampling traces at a certain rate, the collector can apply the same sampling rate to logs, ensuring that traces and logs remain aligned. ### Advantages of OpenTelemetry Logging in Kubernetes By adopting [OpenTelemetry for centralized logging](https://www.checklyhq.com/learn/opentelemetry/logs/), Kubernetes administrators gain several benefits: * **Unified Telemetry Stack**: OpenTelemetry consolidates logging, tracing, and metrics collection, eliminating the need for multiple agents and configuration files. This simplifies management and reduces potential points of failure. * **Flexibility in Log Destination**: Logs can be sent to multiple backends, allowing organizations to maintain control over their data while still meeting specific requirements, such as long-term storage or compliance. * **Improved Efficiency**: OpenTelemetry’s design reduces the CPU and memory overhead of log processing, which is critical in Kubernetes environments where resources are often at a premium. ### Conclusion Centralized logging in Kubernetes is a necessity for maintaining reliable and secure applications. With OpenTelemetry, administrators gain a unified framework for collecting, processing, and exporting logs alongside other telemetry data types. The OpenTelemetry Collector serves as a powerful tool for centralized log management, providing consistency, efficiency, and flexibility to meet the demands of modern, distributed applications. # Kubernetes Events - A Native Resource for State Insight Source: https://www.checklyhq.com/docs/learn/kubernetes/events In Kubernetes, **Events** are a native resource, providing insight into actions and state changes of objects in the cluster. Events are platform-generated messages accessible through the Kubernetes API, often used by operators and users to understand what has happened to various Kubernetes objects. ## Introduction In Kubernetes, **Events** are a native resource, providing insight into actions and state changes of objects in the cluster. Events are platform-generated messages accessible through the Kubernetes API, often used by operators and users to understand what has happened to various Kubernetes objects. While they act similarly to logs, Events differ in that they aren’t usually stored in logging backends; instead, they are stored in Kubernetes’ etcd and retained for a short period, typically around one hour. Events serve as valuable resources for monitoring and debugging, allowing users to quickly gather information about object actions in the cluster. This document introduces Kubernetes Events, their mechanics, and how to use them for effective alerting and monitoring. *** ## Event Mechanics Kubernetes Events can be accessed by querying the Kubernetes API. Each Event describes an action related to a Kubernetes object, such as a Pod, Deployment, or Node. These Events are often labeled with the following attributes: * **Type**: Indicates whether the Event is `Normal` (standard actions) or `Warning` (indicating potential issues). * **Reason**: Short code identifying the type of action. * **Age**: How long ago the Event occurred. * **From**: The source component generating the Event, such as `kubelet`. * **Message**: Describes the action taken or the issue observed. *** ### Example: Viewing Events for a Pod To see Events associated with a specific Pod, use the `kubectl describe` command. Here’s an example output for a Pod named `webapp-5d8dbbd4bb-2lqfs`: ```bash theme={null} $ kubectl describe pod webapp-5d8dbbd4bb-2lqfs Name: webapp-5d8dbbd4bb-2lqfs Namespace: production ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled default-scheduler Successfully assigned production/webapp-5d8dbbd4bb-2lqfs Normal Pulling 45s kubelet, ip-10-0-1-233 Pulling image "webapp:v1.2" Normal Pulled 42s kubelet, ip-10-0-1-233 Successfully pulled image "webapp:v1.2" Normal Created 42s kubelet, ip-10-0-1-233 Created container webapp Warning BackOff 40s kubelet, ip-10-0-1-233 Back-off restarting failed container ``` In this output, each Event shows a timestamp and message about actions related to the Pod, from scheduling to a restart back-off warning. ### Example: Viewing Events for a Namespace To retrieve Events for all resources in a namespace, you can use the `kubectl get events` command: ```bash theme={null} $ kubectl get events -n production LAST SEEN TYPE REASON OBJECT MESSAGE 1m12s Normal Scheduled pod/webapp-5d8dbbd4bb-2lqfs Successfully assigned production/webapp-5d8dbbd4bb-2lqfs 1m10s Normal Pulling pod/webapp-5d8dbbd4bb-2lqfs Pulling image "webapp:v1.2" 1m8s Normal Pulled pod/webapp-5d8dbbd4bb-2lqfs Successfully pulled image "webapp:v1.2" 1m8s Normal Created pod/webapp-5d8dbbd4bb-2lqfs Created container webapp 1m8s Warning BackOff pod/webapp-5d8dbbd4bb-2lqfs Back-off restarting failed container 1m15s Normal SuccessfulCreate replicaset/webapp-5d8dbbd4bb Created pod: webapp-5d8dbbd4bb-2lqfs 1m15s Normal ScalingReplicaSet deployment/webapp Scaled up replica set webapp-5d8dbbd4bb to 1 ``` ## This output shows Events from the namespace, covering various resources, such as Pods, ReplicaSets, and Deployments, with some Events indicating a container restart issue for further investigation. ## Monitoring and Alerting with Events Events provide real-time insights, allowing operators to monitor Kubernetes activity. Although not commonly used for direct automation, some approaches leverage Events for alerting and monitoring: ### Event Exporters An **Event exporter** can expose Events as metrics to external monitoring systems. For example, Prometheus can scrape metrics derived from Events, enabling alerting based on conditions like frequent container restarts or persistent failures in Pod scheduling. Using an Event exporter requires deploying a tool that collects Events and converts them into Prometheus metrics. This setup enables teams to build custom alerts and dashboards, adding proactive monitoring capabilities that go beyond Kubernetes’ default Event retention period. *** ## Conclusion Kubernetes Events offer valuable information about the actions and status of objects within a cluster. With Events, operators gain a straightforward way to monitor and troubleshoot Kubernetes activities. When integrated with external monitoring solutions, Events can also play a significant role in proactive alerting and incident management. # Introducing klog - The Kubernetes Logging Library Source: https://www.checklyhq.com/docs/learn/kubernetes/klog Meet klog, the Kubernetes-specific logging library designed to provide structured and reliable logging across Kubernetes components. In this post, we’ll dive into klog’s purpose, key features, and how it’s transforming logging in Kubernetes. With clusters consisting of multiple nodes and applications, maintaining clear and consistent logs is crucial. Enter klog, the Kubernetes-specific logging library designed to provide structured and reliable logging across Kubernetes components. In this post, we’ll dive into klog’s purpose, key features, and how to get started. ### What is klog? klog is the logging library used within Kubernetes to manage logs across its various components, such as the API server, kubelet, and controller manager. Originally based on Google’s `glog`, klog introduces structured logging and integrates with Kubernetes architecture. As Kubernetes clusters scale, klog’s ability to standardize log output is vital for efficient log collection, storage, and analysis. Starting with Kubernetes v1.19, klog adopted structured logging to make log entries easier to parse, search, and analyze. This change enables administrators to move away from unstructured plain text logs, which often require custom parsing and lead to inconsistencies across components. Structured logs also allow for better integration with logging tools like Elasticsearch, Fluent Bit, and Grafana, making it easier to monitor, alert, and troubleshoot. ### Why klog Matters for Kubernetes Logging In Kubernetes, logs are generated by numerous components, each responsible for specific functions. For example: * **API Server Logs**: Record all interactions with the Kubernetes API. * **Kubelet Logs**: Provide insights into pod management, resource usage, and node health. * **Controller Manager Logs**: Capture information about state reconciliation and resource control. Managing these logs without a consistent format can quickly become overwhelming. klog solves this issue by providing a structured approach to logging, allowing log aggregation and analysis tools to work more effectively. By adopting structured logging, Kubernetes improves observability, making logs easier to query and analyze. ### Key Features of klog 1. **Structured Logging with Key-Value Pairs**: klog enables structured logging by supporting key-value pairs, allowing log entries to be machine-readable and easily parsable. This structure simplifies log queries, as logs with consistent fields can be filtered and aggregated efficiently. 2. **JSON Support**: With JSON as the standard format for structured logs, klog makes it easier to ingest logs into centralized logging solutions. JSON logs are widely supported by logging tools, making it simple to integrate Kubernetes logs with existing monitoring systems. 3. **Enhanced Methods for Consistent Logging**: klog provides new methods (`InfoS` and `ErrorS`) for logging informational and error messages in a structured format. These methods standardize log output across components, reducing inconsistencies and improving the readability of logs. 4. **Seamless Integration with Kubernetes Components**: klog is embedded in the core of Kubernetes, meaning it’s natively supported by key system components. This allows for out-of-the-box structured logging across all critical areas of a Kubernetes cluster. ### klog vs. Glog: Why the Shift? Kubernetes previously relied on `glog`, but as the system grew, the limitations of `glog` became apparent. Glog’s lack of structured logging meant that logs were generated as plain text, making them hard to parse and analyze. With structured logging increasingly becoming the standard in observability, Kubernetes needed a library that could support a consistent format across components. klog, with its JSON support and key-value pair structure, was developed to meet these needs. Structured logging also allows Kubernetes to integrate more easily with third-party monitoring tools, as JSON logs are more compatible with Elasticsearch, Grafana, Prometheus, and other tools that rely on consistent data formatting. ### Key klog Methods: `InfoS` and `ErrorS` To facilitate structured logging, klog introduced the `InfoS` and `ErrorS` methods. These methods replace traditional logging functions by enforcing a structured, key-value format. ### 1. **InfoS**: Log Informational Messages The `InfoS` method captures general information in a structured format. This method requires a message followed by key-value pairs, where each key represents a field, and the associated value provides context. ```go theme={null} klog.InfoS("Pod status updated", "pod", "kube-dns", "status", "ready") ``` This example logs an informational message, specifying the pod name (`kube-dns`) and its current status (`ready`). ### 2. **ErrorS**: Log Error Messages The `ErrorS` method is used for error logging. It requires an error as the first parameter, followed by a descriptive message and key-value pairs that provide context for the error. ```go theme={null} klog.ErrorS(err, "Failed to update pod status", "pod", "kube-dns") ``` In this example, `ErrorS` logs an error with a detailed message and relevant pod information. This structure makes it easier to track down issues within specific pods or components. ### Example: JSON Output with klog klog’s structured approach outputs logs in JSON format, making it easier to parse and analyze. Here’s what a typical JSON log entry might look like when generated by klog: ```json theme={null} { "ts": "2024-11-07T12:00:00Z", "level": "info", "msg": "Pod status updated", "pod": "kube-dns", "status": "ready", "namespace": "kube-system" } ``` Each log entry contains key-value pairs for essential information like timestamp, log level, message, pod name, and status, making it easy for log management tools to filter and analyze the data. ### Benefits of Structured Logging with klog 1. **Improved Observability**: Structured logging makes it easier to monitor Kubernetes environments by providing consistent, easily searchable logs. 2. **Efficient Log Analysis**: By outputting logs in JSON, klog supports efficient parsing and querying, which is essential when working with large volumes of data. 3. **Enhanced Integration**: JSON logs integrate seamlessly with observability tools, allowing Kubernetes administrators to visualize and analyze logs more effectively. 4. **Reduced Parsing Complexity**: Structured logs eliminate the need for custom parsing, as log fields are consistent across entries, enabling more straightforward log processing. ### Best Practices for Using klog in Kubernetes 1. **Use Key-Value Pairs Consistently**: Define standard fields for commonly logged events to keep logs consistent across components. For instance, use `"pod"`, `"namespace"`, and `"status"` consistently in pod-related logs. 2. **Set Log Levels Appropriately**: Use log levels (`InfoS` for informational logs and `ErrorS` for errors) to differentiate critical information from general updates. This ensures that logs are meaningful and that critical events are easy to identify. 3. **Monitor Log Performance**: Structured logging can add overhead, so monitor klog’s impact on performance, particularly in high-traffic clusters. 4. **Integrate with Log Management Tools**: klog is compatible with log management systems that support JSON, so integrate it with tools like Fluent Bit, Prometheus, and Grafana to visualize and analyze logs effectively. ### Implementing klog in Your Kubernetes Environment To enable structured logging with klog, Kubernetes administrators can configure components to use `InfoS` and `ErrorS` methods and output JSON-formatted logs. This setup standardizes logs, improving the efficiency of log aggregation and analysis pipelines. For a fully integrated solution, combine klog with logging agents like Fluent Bit or Fluentd. These agents can collect logs from across the Kubernetes cluster, parse them, and forward them to centralized storage or monitoring systems. ### Future of klog and Structured Logging in Kubernetes The move toward structured logging reflects a larger industry trend towards improving observability across distributed systems. With features like contextual logging and enhanced compatibility with monitoring tools, klog will continue to improve the reliability and manageability of Kubernetes logging. ### Conclusion klog integration into Kubernetes components and compatibility with JSON format makes it a necessary tool for monitoring and troubleshooting in distributed environments. By using klog’s structured logging features, Kubernetes administrators and developers can improve observability, streamline log analysis, and ensure that their logs remain accessible and actionable across the entire cluster. # Logging in Kubernetes - Best Practices Source: https://www.checklyhq.com/docs/learn/kubernetes/logging-best-practices Monitoring logs effectively in a Kubernetes environment requires tools that centralize and streamline log management. Logging in Kubernetes can be a challenge, especially when dealing with data across a distributed, containerized system. To keep your logs efficient and actionable, it's essential to understand logging best practices, from handling diverse formats to managing back pressure effectively. Here are the best practices for Kubernetes logging, focused on tools, performance, and scalability. ### 1. Define Your Logging Goals The main goal of logging isn’t just to collect data—it’s to analyze it and gain insights into your system's health and performance. Collecting excess logs can create noise, so start by defining clear goals. Consider which data is essential for troubleshooting, compliance, and performance monitoring, and focus on collecting only that information. This targeted approach helps reduce unnecessary log volume and keeps analysis efficient. ### 2. Centralize and Standardize Logging Kubernetes applications can produce logs in various formats and locations. Some applications write to standard output, while others log to specific files. In addition, different applications, such as NGINX, syslog, or even Windows services, may log in entirely different formats. Use a centralized logging system to consolidate all logs into a single view. A common tool for this purpose is **Fluent Bit**, which is lightweight, supports various inputs and outputs, and has a plugin system that allows it to integrate with numerous backends. Fluent Bit’s flexibility makes it a solid choice for aggregating logs from multiple sources and delivering them in a standardized format, such as JSON. ### 3. Structure Your Logs Unstructured logs (raw text logs) can be difficult to parse and analyze, especially at scale. Structuring logs, ideally in a key-value format, makes it easier to query and process them. Consider using structured logging formats like JSON to provide clear, consistent fields (e.g., timestamp, log level, message). Fluent Bit can handle log parsing and transformation to convert unstructured logs into a structured format. For example, rather than storing a plain log message, structure it to include relevant metadata: ```json structured-log-example.json theme={null} { "timestamp": "2024-05-01T12:00:00Z", "level": "error", "service": "user-auth", "message": "Authentication failed", "user_id": "12345", "ip_address": "192.168.1.1" } ``` ### 4. Enrich Logs with Contextual Information In distributed systems, it’s often hard to pinpoint where an issue occurred based on logs alone. Enrich your logs with additional metadata, such as the Kubernetes namespace, pod name, and container ID. This metadata provides valuable context that makes debugging easier. Fluent Bit can automatically add Kubernetes metadata to logs, providing instant visibility into which parts of the system are generating specific logs. ### 5. Filter and Reduce Log Volume Logging everything can quickly lead to storage and performance issues, especially with high log volumes. It’s essential to filter out non-critical logs, such as verbose debug logs, to reduce unnecessary data. Implement filters to retain only the logs you need, based on severity (e.g., only errors and warnings in production environments). In Fluent Bit, you can apply filters to drop debug-level logs in production. For instance: ```yaml fluent-bit-filter.conf theme={null} [FILTER] Name grep Match * Exclude log_level debug ``` ### 6. Handle Sensitive Data Securely Logs can sometimes contain sensitive information, such as user IDs or payment data. To prevent exposing sensitive data, redact or mask it in logs. Fluent Bit supports redaction and can modify logs to obscure sensitive data before it’s stored or transmitted. Example: Use a Lua script or regular expressions in Fluent Bit to mask sensitive information like credit card numbers. ```yaml fluent-bit-redaction.conf theme={null} [FILTER] Name modify Match * Add card_number "xxxx-xxxx-xxxx-1234" ``` ### 7. Use Logs-to-Metrics Conversion Some logs contain high-frequency, repetitive events that might be more useful as metrics rather than logs. Converting these logs into metrics can improve performance and reduce storage needs. For instance, tracking the number of failed authentication attempts is often more efficient as a metric. Fluent Bit can convert log entries to metrics, which can then be visualized and analyzed in tools like Prometheus. This approach reduces storage costs and enhances visibility for events that are better tracked over time. ### 8. Manage Back Pressure and Buffering In high-throughput environments, the risk of back pressure increases, especially when logs are sent to remote endpoints with limited processing capacity. Fluent Bit provides buffering options, including in-memory and file-based buffers, to handle spikes in log traffic. Using a file-based buffer is generally more reliable for production, as it prevents data loss if in-memory buffers are exhausted. Configure Fluent Bit with file-based buffering for resilience: ```yaml fluent-bit-output.conf theme={null} [OUTPUT] Name es Match * Host elasticsearch.example.com Port 9200 Buffer_Chunk_Size 1M Buffer_Max_Size 5M ``` ### 9. Monitor Log Pipeline Health Once your logging infrastructure is in place, monitor its health to ensure consistent performance. Metrics like input and output rates, dropped logs, and processing latency can reveal bottlenecks and prevent data loss. You can use Prometheus to monitor Fluent Bit’s internal metrics and set up alerts for issues like increased log drop rates or high memory usage. Regular monitoring helps you catch problems early and ensures that your logging pipeline remains effective. ### 10. Plan for Scale Kubernetes clusters grow over time, so it’s essential to design your logging solution to scale with your application. Ensure your log storage backend can handle increased volumes, and consider using Fluent Bit’s load-balancing capabilities if you have multiple log destinations. You can scale Fluent Bit horizontally by running multiple instances in Kubernetes and balancing log traffic across them. This architecture helps maintain stability and reduces the risk of data loss during peak loads. ### Conclusion Logging in Kubernetes involves more than just collecting logs; it requires a thoughtful approach to structuring, filtering, and managing data at scale. With the right tools and practices, you can create a robust logging pipeline that supports your troubleshooting and monitoring needs without overwhelming your infrastructure. Following these best practices can lead to a more effective, scalable, and cost-efficient logging system in Kubernetes. # Learn How to Observe and Monitor your Kubernetes Environment with Open Source Tools Source: https://www.checklyhq.com/docs/learn/kubernetes/monitoring-challenges Managing Kubernetes in production is no small feat, especially when it comes to monitoring and observability. Managing Kubernetes in production is no small feat, especially when it comes to monitoring and observability. As modern applications grow in complexity, traditional methods of monitoring, limited to CPU and memory usage, often fall short. In this article, we explore the unique challenges of monitoring Kubernetes environments and the open-source tools that can improve visibility and control. ### The Layers of Complexity in Kubernetes Starting with Kubernetes can be straightforward—deploying a service, setting up replicas, and configuring basic settings. But as deployment scales, the complexity grows significantly. Organizations must manage stateful services, custom resource definitions (CRDs), networking, and more, which are essential to production-grade Kubernetes. Even experienced administrators can find these layers challenging, especially as each layer adds new demands and potential points of failure. To aid with this complexity, the Cloud Native Computing Foundation (CNCF) offers a trail map of projects suited to various aspects of Kubernetes management, from containerization to continuous integration, service discovery, and, importantly, observability. ### The Core of Observability: Logs, Metrics, and Traces Observability in Kubernetes focuses on three pillars: logs, metrics, and traces. Each plays a distinct role in providing a clear picture of an application’s health and performance. ### 1. Centralized Logging In dynamic, cloud-native environments, centralized logging is essential. Logs help trace what happened in each instance, and they provide necessary details for compliance, especially in regulated industries like finance, which often require logs to be retained for years. With centralized logging, administrators avoid the pitfalls of ephemeral logging. For example, if Kubernetes nodes are updated or workloads shift, logs can disappear. Tools like Loki, an open-source log aggregation tool, enable centralized logging by collecting logs across services and making them accessible in tools like Grafana. This setup ensures that logs are available from a central point, reducing the need to search individual nodes or pods. ### 2. Metrics Collection Metrics provide a big-picture view of resource consumption and application health over time. Unlike logs, which capture individual events, metrics are aggregated and are ideal for spotting trends, such as CPU or memory usage spikes, that indicate resource needs or scaling requirements. Prometheus is a widely used tool in Kubernetes environments for metrics collection. Combined with Grafana, which provides visualization, administrators can access real-time dashboards that show cluster CPU usage, memory utilization, and other metrics. These insights are invaluable in optimizing resources and setting up auto-scaling rules to adjust to application demand. ### 3. Distributed Tracing With microservices architectures, distributed tracing is crucial. It tracks the journey of requests across services, helping identify bottlenecks and pinpointing where failures occur. This is particularly helpful in production, where diagnosing issues quickly is critical to maintaining uptime. Jaeger, a distributed tracing tool, integrates with OpenTelemetry to capture trace data across microservices, making it possible to follow a request’s entire journey and spot where time is spent or where errors occur. This visibility is essential for optimizing performance and understanding how services interact. ### OpenTelemetry: A Standardized Approach OpenTelemetry is a vendor-neutral standard that simplifies observability by consolidating data from multiple sources. It offers a single framework for instrumenting code, collecting, and exporting logs, metrics, and traces, reducing the need to install multiple agents or integrate separate SDKs. OpenTelemetry allows switching backends without major reconfiguration, avoiding vendor lock-in. ### Challenges in Kubernetes Observability Monitoring Kubernetes comes with its own set of challenges: * **Ephemeral Logs**: Logs can be lost if they aren’t centrally stored, especially with Kubernetes nodes being replaced during upgrades. * **Vendor Lock-In**: Proprietary monitoring tools can lead to lock-in. Open-source and standards-based tools like OpenTelemetry and Grafana help avoid this. * **High Setup Overheads**: While the ecosystem offers many observability tools, setting them up and maintaining them can be complex. ### Tools to Consider Here are a few recommended tools for Kubernetes observability: * **Loki for log aggregation** * **Prometheus for metrics collection** * **Jaeger for distributed tracing** * **Grafana for a unified view of logs, metrics, and traces** ### Conclusion Kubernetes has expanded our operational flexibility and reliability. In so doing, the number of microservices has grown, and the result has been larger observability challenges, but the right tools and strategies make it manageable. By embracing open standards like OpenTelemetry and relying on proven open-source tools, organizations can improve observability, gain insight into application performance, and respond effectively to issues. # Introduction to Node Problem Detector for Kubernetes Source: https://www.checklyhq.com/docs/learn/kubernetes/node-problem-detector The Node Problem Detector (NPD) is a Kubernetes tool for monitoring the health of nodes. Designed to identify and report a range of node issues, NPD aims to maintain cluster stability by detecting faulty nodes and marking them as unschedulable The **Node Problem Detector (NPD)** is a Kubernetes tool for monitoring the health of nodes. Designed to identify and report a range of node issues, NPD aims to maintain cluster stability by detecting faulty nodes and marking them as unschedulable, so that workloads do not get assigned to potentially unstable environments. NPD's purpose is to make sure workloads only run on healthy nodes, minimizing disruptions from faulty node behavior. ### How Node Problem Detector Works Node Problem Detector operates by running as a DaemonSet on each node in the Kubernetes cluster. It monitors system logs and checks for specific conditions or failures that could compromise the node's stability. Once NPD detects an issue, Kubernetes marks the node as "unschedulable." This status informs the cluster's scheduling process to exclude the node, effectively reducing the cluster’s available capacity without impacting running workloads. ### Remediation and Challenges While NPD does identify and isolate nodes with issues, automatically remediating these problems can be more challenging. Some of the common issues, like a read-only root filesystem, may point to deeper hardware faults or configurations that may need manual intervention. NPD's role is limited to problem detection; it doesn’t solve the underlying issues, which often require hardware fixes or support from system constructors, especially in physical environments. In virtualized environments, such as on cloud-managed Kubernetes clusters, nodes flagged by NPD can often be drained and replaced. This is common in platforms like Azure Kubernetes Service (AKS), where a failing node can be terminated, and a new one created by the autoscaler. However, simply replacing nodes might not address the underlying cause, especially if it stems from node misconfiguration or application-related issues. ### Use Cases and Limitations NPD's effectiveness shines in physical clusters where hardware inconsistencies can cause unexpected issues. Having NPD exclude problematic nodes prevents further complications and can be a stopgap until the hardware can be inspected or replaced. In virtual environments, NPD's role can sometimes be limited, as issues tend to arise more from configuration or workload setup, both of which should ideally be resolved in staging before hitting production. In summary, Node Problem Detector acts as a health monitoring and early-warning system, isolating problematic nodes to protect workload stability. It’s especially valuable in detecting issues in physical nodes where hardware problems may be more common. However, for automated remediation in virtual clusters, the approach is usually to replace the node entirely, although this doesn’t address root causes directly. ## How to Enable the Node Problem Detector To enable **Node Problem Detector (NPD)** on your Kubernetes cluster, follow these steps to deploy and configure it as a DaemonSet. This deployment will ensure that NPD runs on each node, actively monitoring for issues and reporting them as node conditions and events. ### 1. Deploy Node Problem Detector as a DaemonSet To deploy NPD across your nodes, apply the official NPD DaemonSet configuration provided by the Kubernetes project: ```bash theme={null} kubectl apply -f https://k8s.io/examples/debug/node-problem-detector.yaml ``` This command downloads and applies a predefined DaemonSet configuration for NPD, ensuring it runs on each node in your cluster. The configuration file contains necessary setup for resource limits, logging, and permissions needed by NPD to function properly. ### 2. Verify the Deployment After deploying the DaemonSet, confirm that NPD is running on each node: ```bash theme={null} kubectl get daemonset -n kube-system node-problem-detector ``` This command checks the status of the NPD DaemonSet in the `kube-system` namespace, where it’s typically deployed. Look for the `DESIRED`, `CURRENT`, and `AVAILABLE` fields to confirm that the DaemonSet is running on all nodes as expected. ### 3. Check Node Problem Detector Logs To see what issues NPD is detecting, check its logs. This can be useful for troubleshooting or confirming that NPD is actively monitoring node health: ```bash theme={null} kubectl logs -l k8s-app=node-problem-detector -n kube-system ``` This command fetches logs from all pods in the NPD DaemonSet, allowing you to review any node issues that NPD has detected. ### 4. Configure Node Problem Detector (Optional) The default configuration may not cover all specific needs of your cluster. For example, you can customize what types of problems NPD detects by modifying the configuration file or using custom plugins. To edit NPD’s configuration, download the DaemonSet YAML file, make necessary changes, and reapply it: ```bash theme={null} kubectl apply -f /path/to/your-modified-node-problem-detector.yaml ``` Refer to the official [Node Problem Detector documentation](https://github.com/kubernetes/node-problem-detector) for available configuration options and plugins. Customizing NPD lets you tailor detection parameters to your environment, making it more effective in identifying relevant node issues. ### 5. Confirm NPD’s Functionality To ensure NPD is operating correctly, simulate a node condition or trigger an event that it should detect. Verify that Kubernetes marks the node as "unschedulable" or logs an appropriate event when NPD reports the condition. ## Conclusions The Node Problem Detector is a core piece of Kubernetes functionality, but the exact path to follow once a node is marked unschedulable is left up to the user. Building alerting, or remediation, on the problem detector is a bespoke problem for operators. # Introduction to Kubernetes Operators - scraping metrics with Prometheus Source: https://www.checklyhq.com/docs/learn/kubernetes/operators-metrics Operators in Kubernetes enable custom automation by encapsulating specific operational knowledge within Kubernetes itself. They function as custom controllers that manage complex applications and provide features like automated deployment, scaling, and management. Operators in Kubernetes enable custom automation by encapsulating specific operational knowledge within Kubernetes itself. They function as custom controllers that manage complex applications and provide features like automated deployment, scaling, and management. Operators work by observing resources and adjusting Kubernetes state based on custom logic, extending beyond standard deployment and service management. With operators, you can manage resources beyond the built-in types like Pods and Services. For example, creating an operator to deploy a database that automatically scales, backs up, and restores adds significant automation capability to Kubernetes. #### What is Kubebuilder? Kubebuilder is a development framework for building Kubernetes operators using the Go programming language. It provides project scaffolding, resource definitions, and controller logic to simplify operator creation. In short, Kubebuilder allows developers to create CRDs (Custom Resource Definitions) and controllers, which watch Kubernetes resources and act upon changes. #### Setting Up a Project with Kubebuilder The typical process starts by setting up a basic project structure using the `kubebuilder init` command. This scaffolds the folders and files necessary for an operator project. The following commands initialize a project and create a custom API for a simple application: ```bash theme={null} kubebuilder init --domain example.com --repo github.com/example/my-operator kubebuilder create api --group app --version v1 --kind WebApp ``` This creates a project structure that includes an API definition for `WebApp`, a custom resource that will deploy and manage a Kubernetes Deployment and Service. #### Adding Metrics to the Operator For observability, we can integrate Prometheus metrics into the operator. In Kubernetes, Prometheus can scrape data from custom endpoints to provide insights into application behavior. By using the `prometheus.NewCounter` method from the Prometheus Go client, we set up counters to track resource actions, such as the creation and deletion of `WebApp` instances. In the controller’s main loop, you might add code like this to register counters: ```go theme={null} var ( createCounter = prometheus.NewCounter( prometheus.CounterOpts{ Name: "webapp_create_total", Help: "Total number of WebApp creations", }, ) ) func init() { prometheus.MustRegister(createCounter) } ``` Each time the reconciliation loop (the main loop of the controller) handles a creation or update event, the counter increments. #### Deploying Prometheus to Scrape Metrics You must configure Prometheus to scrape the operator’s metrics endpoint. To achieve this, add a scrape configuration to the Prometheus setup, pointing to the operator service endpoint: ```yaml theme={null} scrape_configs: - job_name: 'webapp_operator' static_configs: - targets: ['webapp-operator-service.default.svc.cluster.local:8080'] metrics_path: /metrics scrape_interval: 15s ``` With Prometheus deployed, the operator exposes metrics under the `/metrics` path, which Prometheus regularly scrapes to collect data. #### Running the Operator and Observing Metrics After deploying the operator to a Kubernetes cluster (e.g., a local KIND cluster), we can create custom `WebApp` resources that initiate deployments. By applying YAML files with the custom `WebApp` definitions, developers can observe metrics that Prometheus scrapes and view them in a dashboard like Grafana. For instance, creating or updating a `WebApp` resource increases the counter for created deployments, visible in both the Prometheus and Grafana dashboards: ```yaml theme={null} apiVersion: app.example.com/v1 kind: WebApp metadata: name: sample-webapp spec: replicas: 2 image: nginx:latest port: 80 ``` Upon applying this YAML, Prometheus will capture the increment in creation counters. Similarly, any updates to the deployment reflect in the metrics, making it possible to track deployment status directly from Prometheus. #### Conclusion Operators with custom metrics offer a valuable mechanism for managing Kubernetes resources and observing application behavior in real time. Using Kubebuilder and Prometheus, developers can integrate meaningful metrics into their operators, enabling better monitoring and automation within Kubernetes. This combination of Kubebuilder for operator creation and Prometheus for metric collection allows teams to leverage Kubernetes for more than just basic application hosting—it becomes a powerful automation and observability platform. # Learn How to Observe and Monitor your Kubernetes Source: https://www.checklyhq.com/docs/learn/kubernetes/overview Learn OpenTelemetry with Checkly. Add monitoring to every piece of your stack with the open standards and open-source tools. Learn about monitoring Kubernetes. ## Getting Started Learn the basics of monitoring your Kubernetes cluster. Learn how to instrument your application and get deep insights into your stack. ## Diving Deeper Learn about structured logging in Kubernetes. Learn about key events of the Kubernetes lifecycle. # Structured Logging in Kubernetes - Why it matters and how to get started Source: https://www.checklyhq.com/docs/learn/kubernetes/structured-logging Don't put unstructured blobs of data into your Kubernetes logs! For developers and administrators, adopting structured logging means enhancing observability, streamlining log analysis, and improved system performance. In this post we’ll cover the fundamentals of structured logging in Kubernetes, its practical applications, and best practices for implementing it in your systems. ### Why Kubernetes Logging Matters Logs capture detailed information about cluster activities, from system component interactions to application behavior. However, Kubernetes logs often come from various sources (containers, system components like etcd, API servers, and nodes) and in inconsistent formats. This can make troubleshooting difficult, especially when logs need to be aggregated for analysis. Log aggregation helps centralize logs, but without a standardized format, querying logs effectively becomes challenging. Structured logging addresses this issue by providing a consistent, predictable log format across all components, making it easier to analyze and troubleshoot Kubernetes systems. ### What is Structured Logging? Structured logging means organizing logs into a standardized format, often using key-value pairs. This structure makes it easier to parse, search, and analyze logs, eliminating the need for ad-hoc solutions like regular expressions. Kubernetes structured logging leverages JSON as the standard output format, which is human-readable, widely supported by logging libraries, and efficiently parsed by systems. ### Example of Structured Logging Format A structured log entry in Kubernetes follows a set format: ```json theme={null} { "timestamp": "2024-11-07T12:00:00Z", "level": "info", "message": "Pod status updated", "pod": "kube-dns", "status": "ready", "namespace": "kube-system" } ``` This JSON structure ensures consistency across logs, allowing easy querying and integration with log management tools like Elasticsearch, Prometheus, and Grafana. ### Implementing Structured Logging in Kubernetes To standardize Kubernetes logs, new methods (`InfoS` and `ErrorS`) were introduced in the Klog library. These methods provide a consistent structure for log entries, enabling logs to be exported in JSON format. ### Key Components of Structured Logging 1. **Message and Key-Value Pairs**: Structured log messages include a main message followed by key-value pairs. 2. **Klog Methods**: * `InfoS`: Captures standard informational logs. * `ErrorS`: Captures error logs, with the error as the first parameter, followed by the message and key-value pairs. ### Example Usage ```go theme={null} klog.InfoS("Pod status updated", "pod", "kube-dns", "status", "ready") ``` This produces a log entry with a standardized structure, making it easy to interpret and analyze. ### Why JSON for Structured Logging? JSON was chosen for Kubernetes structured logging because it’s broadly adopted, human-readable, and easily parsed by machines. JSON’s format supports structured data naturally, and tools like `jq` make it straightforward to filter and process JSON logs. By standardizing on JSON, Kubernetes ensures compatibility with most log management and analysis tools, helping reduce friction in log aggregation and processing. ### Performance Benefits of Structured Logging Migrating to structured logging in JSON format has shown performance improvements. Benchmarks indicate that structured logs in JSON format are significantly faster to parse and query than plain text logs. This improved performance is due to the predictability and standardized structure of JSON logs, which can be processed with fewer computational resources. ### Contextual Logging: Adding Context to Logs Contextual logging is an advanced form of structured logging. It allows you to attach contextual data to log entries, which helps in understanding the source of specific log events. This is particularly useful in complex Kubernetes environments where logs from various components need to be correlated. Contextual logging in Kubernetes uses the Go `logger` API, which is designed around structured logging principles and supports attaching additional metadata. This metadata provides context, such as which Kubernetes pod or scheduler plugin a particular log entry is associated with. ### Key Design Decisions in Contextual Logging 1. **Attaching the Logger to Context**: By attaching the logger as a value to the context, functions can use contextual information without requiring explicit parameters. 2. **Retrieving the Logger from Context**: The logger can be retrieved from the context, ensuring that all relevant metadata is accessible across function calls. ### Practical Example of Contextual Logging Imagine a scenario where a developer wants to track which pod and operation a particular log message is associated with. By using contextual logging, they can attach relevant details to the log message, making it easier to filter logs based on pod and operation identifiers. ```go theme={null} logger := klog.FromContext(ctx).WithValues("pod", "kube-dns", "operation", "volume-binding") ``` This will produce logs with a prefix like `kube-dns/volume-binding`, helping the developer quickly locate logs related to specific pods or operations. ### Best Practices for Structured Logging in Kubernetes To maximize the benefits of structured logging, consider these best practices: 1. **Use Key-Value Pairs Consistently**: Ensure that logs across different Kubernetes components use similar key-value pairs for the same types of data. This consistency simplifies queries and analysis. 2. **Limit Log Levels**: Structured logs should focus on meaningful events. Avoid logging verbose data by default to reduce log noise and storage costs. 3. **Avoid Logging Sensitive Information**: Take care not to log sensitive data, such as passwords or personal information. If sensitive data must be logged for troubleshooting, redact it before storing or transmitting logs. 4. **Monitor Log Performance**: Monitor the performance impact of structured logging, especially if you’re logging high volumes of data. Contextual logging can add overhead, so consider trade-offs between logging detail and performance. 5. **Test JSON Parsing**: Ensure that your JSON logs are correctly formatted and parsable. Invalid JSON structures can cause issues in downstream log analysis tools. 6. **Leverage Contextual Logging for Complex Workflows**: Use contextual logging when tracking complex workflows or operations across Kubernetes components. This can simplify debugging by adding relevant context to logs. ### Current Status and Future of Structured Logging Structured logging in Kubernetes is now generally available (GA), and contextual logging is expected to enter beta soon. If you’re interested in contributing, the Kubernetes community has ongoing efforts to enhance structured logging through working groups and regular meetings. ### Getting Started with Structured Logging in Your Cluster 1. **Enable Structured Logging**: Ensure Kubernetes components are configured to use structured logging, ideally in JSON format. 2. **Integrate with Log Management Tools**: Use tools like Fluent Bit, Elasticsearch, and Grafana to collect, store, and analyze your structured logs. 3. **Experiment with Contextual Logging**: Add contextual logging where it provides meaningful insights, such as when debugging complex interactions between components. ### Conclusion Structured logging is a foundational improvement for observability in Kubernetes. By adopting structured and contextual logging, you can make your logs more actionable, improve troubleshooting workflows, and enhance your cluster’s reliability. For developers and administrators, structured logging provides a robust, future-proof solution for managing the complexity of Kubernetes environments. # What is API Monitoring? - Tools, techniques, and examples Source: https://www.checklyhq.com/docs/learn/monitoring/api-monitoring The goal of API monitoring is to determine whether the APIs are functioning as they should and whether they are available and functioning at an optimal level for the other applications and services that rely on them. API monitoring is the practice of evaluating how an API (Application Programming Interface) is performing over time via several different metrics, including verifying availability, verifying correctness (i.e., is the data that is being sent and received correctly?), and measuring performance and asserting that against a performance threshold. The goal is to determine whether the APIs are functioning as they should and whether they are available and functioning at an optimal level for the other applications and services that rely on them. Tracking performance data of APIs, such as response times and variations in performance in the context of different environments, enables you to identify issues with them before customers and other stakeholders do, helping you avoid extended periods of downtime or degraded performance. A diagram of dependencies

## Why is API Monitoring Important? When a web application or website delivers less than optimal performance—or fails—the impact on your business or customer experience can be significant. API monitoring tools can be used to check your mobile, web app, or other APIs for *performance*, *correctness*, and *availability*. * **Performance Monitoring:** Monitoring an API endpoint for response time to requests could involve measuring TCP, DNS, and first byte times. * **Correctness Monitoring:** Checking that an API is returning the correct data payload, authentication, status codes, and headers and validating that any given API is functioning correctly. * **Availability Monitoring:** Whether your API is available from a single location or via multiple points across the globe, availability is a vital monitoring metric. If an API is accessible on a global basis, monitoring that API from various locations across the globe can give you insight into any region-specific issues. A good monitoring solution can provide information on all of the aspects outlined above. Once equipped with this knowledge, teams can take the appropriate steps to both fix any damage done as well as [improve suboptimal APIs](https://blog.dreamfactory.com/8-tips-for-optimizing-an-api/) for long-term stability. More importantly, having an efficient alerting regimen working in concert with your monitoring solution can ensure that the right people are notified of problems as soon as they emerge. Precise and accurate reporting then provides additional insights necessary for a successful investigation into the problem. All of this translates into faster time to recovery, fewer problems being encountered by the end-user, and helps mitigate revenue loss. **Get Started for Free:** [Try Checkly for API Monitoring Today](https://app.checklyhq.com/signup?utm_medium=blog\&utm_source=organic\&utm_term=get-started-section) ## Benefits of API Monitoring API monitoring offers several important benefits that enhance the efficiency, security, and performance of applications and systems. It enables organizations to proactively monitor and detect potential issues, ensuring smooth operations. Here are the key advantages of API monitoring: | **Benefits** | **Description** | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Improved Insights and Performance | Continuous API monitoring provides valuable insights to enhance operations. It identifies bottlenecks, optimizing response times and resource utilization. It also tracks API dependencies, ensuring seamless performance. Moreover, monitoring ensures high availability and detects anomalies for prompt issue resolution. | | Enhanced API Performance | Thanks to API monitoring, you can track performance over time, which then enables you to reduce latency, optimize scalability, detect errors, and optimize caching. | | Improved API Metrics | Monitoring elevates API metrics, including uptime and [SLAs](https://www.checklyhq.com/blog/sla-slo-sli/). It tracks API uptime, ensuring uninterrupted service, and measures performance against SLAs. By monitoring key metrics like response times and error rates, deviations from agreed standards are promptly addressed. | | API Responses Accuracy | Ensuring the accuracy of API responses is crucial. Monitoring validates data against predefined rules, detects errors, and sets up alert notifications for quick actions. It also provides performance insights related to accuracy. | Incorporating API monitoring into your software development workflow is instrumental in optimizing operations, enhancing user experiences, and making data-driven decisions for continuous improvements. ## API Monitoring Use Cases Each team will have its own primary use case for API monitoring, but the most common use cases will usually be one of the following. * **Endpoint Uptime Monitoring**: The simple monitoring of 'are we returning 200 responses and valid objects?' Regular pinging of API endpoints to ensure they are operational, which is critical for service availability. * **Performance Benchmarking**: Tracking the response times of APIs to assess and maintain optimal performance levels. A more analog measurement than uptime monitoring. Seeing that your response times are slipping can indicate underlying infrastructure issues or technical debt in code performance. * **Global Availability Checks**: Testing API availability and responsiveness from various global locations to ensure consistent user experiences worldwide. Especially when you have differences in data sovereignty or other cloud architecture differences, this becomes a key check. When I was in Enterprise Services, I always checked our global status first when a high-level customers complained of outages, as geo differences were the most common cause of a hidden issue. * **Versioning and Deployment Validation**: Using [synthetic monitoring](https://www.checklyhq.com/blog/what-is-synthetic-monitoring/) post-deployment to confirm that the API functions as expected, catching issues beyond the scope of unit or integration tests. It's even possible to hook these results directly to a canary deployment system! * **SLA Compliance Monitoring**: Measuring API performance and uptime against Service Level Agreements to maintain customer trust and transparency. This SRE's best tool for tracking SLA compliance is an effective API monitoring tool. ## Do You Need to Monitor APIs Constantly? With applications relying heavily on APIs for their functioning, monitoring APIs on a regular, consistent basis becomes a necessary practice in any software development and maintenance project. Companies that are committed to ensuring that end-users have a positive and smooth experience with their product often take advantage of the best API monitoring tools and services to notify them of any issues before customers encounter them. Around-the-clock, 24/7 API monitoring is essential to catch any issues with APIs that could negatively impact the health of applications. Without API monitoring in place, businesses risk the development of gaps in their knowledge of the application’s status, which can have severe consequences for the stability of the application, and the end-users’ experience of it. Another compelling reason for 24/7 monitoring is that there are multiple points of potential failure in API operations. An update to the API itself might introduce a bug, which leads to downtime or performance degradation, but the same might happen due to infrastructure reasons (e.g., maybe a server or a hosted cloud instance is overloaded). API monitoring should ideally be incorporated into a team’s larger API testing and monitoring efforts so that they can have a comprehensive view and understanding of the application performance and its various components at any given point in time. ## How to Monitor Your APIs Now that we have established what API monitoring is and why businesses should adopt it as part of their development process, here are some tips to get started. The first step is to identify which APIs are currently being employed by your system, including both internal APIs as well as external third party APIs. Understanding the exact role they play in the development of your application is crucial to knowing which metrics need to be monitored and tested. Important metrics such as [API response time](/docs/detect/synthetic-monitoring/api-checks/response-limits), availability, and data correctness are a few that you will be interested in keeping an eye on. Requests are sent to the API, whose responses are evaluated in terms of speed, availability, and correctness. In case the response received does not meet the standards laid out for it, the [API check](/docs/detect/synthetic-monitoring/api-checks/overview) registers an error. It is common for monitors to send a second request to that API in the event of a failure. If the API is unreachable or its response is once again inadequate, an alert is triggered, and pre-defined developers and API-providers are informed about the situation. The next step is to choose an API monitoring tool that will best support your needs. A good monitoring tool will [run checks](/docs/detect/synthetic-monitoring/overview) across all your APIs frequently and be able to check for various metrics across criteria such as performance and correctness. With [Checkly](https://app.checklyhq.com/signup?utm_medium=blog\&utm_source=organic\&utm_term=get-started-section) you can establish a high frequency of checks, which allows you to observe the status of the target system with greater granularity. For example, a critical API endpoint could be monitored every ten seconds to ensure that any issues are caught immediately rather than waiting on a slower monitoring frequency. Additionally, Checkly enables you to monitor APIs from over 20 locations to ensure that they operate smoothly globally. Checkly’s [API monitoring tool](https://www.checklyhq.com/product/api-monitoring/) is an optimal solution for companies keen to develop and integrate an effective API monitoring flow into their existing development and maintenance processes. Checkly enables you to streamline the API monitoring process. You can start by creating or importing HTTP requests for the endpoints you want to monitor, then specify assertions against the response (and response time). There are ample opportunities to customize [setup and teardown](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown) [using scripts](/docs/guides/setup-scripts-for-apis) to fetch OAuth tokens or help clean test data. Checkly has data centers in twenty locations worldwide, from which it can run API checks up to every ten seconds. This is crucial to get a comprehensive and accurate picture of the health of your APIs. Finally, depending on your workflow, [Checkly also integrates with services such as Slack](/docs/communicate/alerts/channels), email, and Pagerduty to notify and alert you of any incidents. ## API Monitoring as Code You can also use Checkly with Infrastructure-as-Code tools like Terraform and Pulumi, or [the official Checkly CLI](https://www.checklyhq.com/cli/overview), to extend the benefits of IaC to collaborative API monitoring, having your monitoring defined right next to your existing code and fitting your existing workflows. * **Get Started for Free:** [Try Checkly for API Monitoring Today](https://app.checklyhq.com/signup?utm_medium=learn\&utm_source=organic\&utm_term=get-started-section) ## How to Set Up API Monitoring Using Checkly Setting up API monitoring using Checkly is straightforward, with all its sufficient resources, features and guides. If you’re able to build a working API, then you should find it easy to follow the steps below and start monitoring your APIs. Let’s see how to set up API monitoring using Checkly: ### Step 1: Sign up for a Checkly Account To get started, head over to the [Checkly website](https://app.checklyhq.com/signup) and sign up for an account if you don’t have one already. A diagram of dependencies

By default, you have access to a free Team trial which expires after 14 days, so in order to continue enjoying the advanced features, you can [subscribe to a pricing plan](https://www.checklyhq.com/pricing/) depending on your needs and requirements. ### Step 2: Create a New Check Once you've signed up and logged in, you can proceed to a [Browser Check](/docs/detect/synthetic-monitoring/browser-checks/overview), API Check or use the CLI script. Let’s play around with the API Check. On your dashboard, click the "API Check" tab (you can toggle between tabs). You should see an interface like this: A diagram of dependencies

You can either check your API, by entering the URL and selecting the HTTP method (GET, POST, PUT, DELETE) the URL accepts, import using a cURL command or import from Swagger or OpenAPI. For instance, using cURL, make a request to your API in your browser, go to the developer tools and find the particular request you want to monitor. Let’s use a request from [https://fakestoreapi.com](https://fakestoreapi.com/), and copy the request from the Chrome developer tools as cURL just like the image below: the Checkly interface

Here is what the cURL looks like: ```bash theme={null} curl 'https://fakestoreapi.com/products/1' \ -H 'authority: fakestoreapi.com' \ -H 'accept: */*' \ -H 'accept-language: en-US,en;q=0.9' \ -H 'dnt: 1' \ -H 'if-none-match: W/"16c-MMdrqY6N0sTiefLdsgtBej9eunY"' \ -H 'referer: https://fakestoreapi.com/' \ -H 'sec-fetch-dest: empty' \ -H 'sec-fetch-mode: cors' \ -H 'sec-fetch-site: same-origin' \ -H 'user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1' \ --compressed ``` Go back to your Checkly dashboard and click the “Import a cURL command” button. A dialog box will pop up. Paste the cURL you copied in the input field and then click “Import cURL”. Now, you should have a more detailed interface containing information and settings for the API request. A diagram of dependencies

### Step 3: Configure the Request Settings In this step, you need to configure various settings for your check. Start by giving your check a name that describes its purpose or functionality. You can also add optional tags to categorize your checks. Next, since the cURL command already specified the request details such as the HTTP method (GET, POST, PUT, DELETE), URL, headers, body/query parameters and authorization (tokens or API keys), the Checkly UI enables you to change/update these details. ### Step 4: Set Up Monitoring Frequency Monitoring frequency determines how often Checkly will send requests to your API endpoint to check its availability and performance across multiple or selected data centers under the “SCHEDULING & LOCATIONS” tab. the Checkly interface

You can choose from various intervals like every minute or custom intervals based on your needs. You can also add assertions and limits to validate specific response criteria like status codes or response time. ### Step 5: Define Alert Channels Alert channels allow you to receive notifications when issues or errors are detected during API monitoring. Checkly supports multiple alert channels including email, phone calls, SMS, Slack, Discord, Telegram, PagerDuty, Webhooks and more. Configure the desired alert channels and provide relevant contact information. ### Step 6: Review and Save Before saving your check configuration, it's essential to review all the settings and ensure they accurately meet your needs and requirements. Double-check the request details, assertions, monitoring frequency, and alert channels. Once you're satisfied, click the green “Save Check” button at the right-top corner. ### Step 7: Monitor and Analyze With your API check set up, Checkly will start monitoring your APIs according to the defined frequency. You can now sit back and let Checkly do its job. Checkly provides comprehensive dashboards and reports to help you analyze the performance and health of your APIs. Utilize these insights to identify potential bottlenecks, optimize response times, and ensure overall reliability. From our example in this article, Checkly is monitoring the API from our Fake Store API. the Checkly interface

By following these steps and utilizing Checkly's intuitive interface and features, you can easily set up a comprehensive API monitoring strategy. Implementing a reliable monitoring solution like Checkly helps you stay proactive in identifying and resolving issues before they impact your users or business. ## How to Choose an API Monitoring Tool: Key Parameters When choosing the best API monitoring tools, it's essential to consider several key parameters to select the right one among the many available options. These parameters will help you choose a tool that meets your specific requirements and ensures the effective monitoring of your APIs. Let's explore these parameters in detail. ### API monitoring usability **Usability** is paramount when selecting API monitoring tools. Opt for a tool with an intuitive interface, customizable reports, and streamlined setup. Real-time alerts can be a game-changer, providing instant notifications through various channels like email, Slack, or SMS to keep you informed and allow prompt action. Additionally, look out for monitoring tools betting on code-first configuration with a [Monitoring as Code approach](https://www.checklyhq.com/guides/monitoring-as-code/) to control and update your monitoring setup and application in the same process. ### Valuable API monitoring metrics **Metrics** are essential for evaluating API performance. Here are some key metrics to consider when monitoring APIs: * **Response Time**: This metric measures the time it takes for API transactions to take place. By monitoring the response time, you can identify if there are any delays or performance issues. * **Error Rate**: The error rate metric helps you determine the percentage of requests that encounter errors. Monitoring this metric enables you to proactively identify and resolve any issues that may impact the reliability of your APIs. * **Throughput**: Throughput measures the number of requests processed by an API within a given time frame. Monitoring throughput can help you determine if your API can handle the expected load and scale appropriately. * **Latency**: Latency is the delay between sending a request and receiving a response. By monitoring latency, you can identify any delays in data transmission and optimize your API's performance. Tools like Checkly offer real-time data collection and visualization, enabling data-driven decisions for improving API performance monitoring. ### Essential monitoring integrations **Integration** ensures seamless operations and efficient error detection. Choose a tool with robust integration capabilities, like Checkly, which [integrates with tools](/docs/communicate/alerts/channels) like Slack, PagerDuty, and Jira. It also offers a REST API for automation and integration into existing workflows. ### Resources and technical support **Support** is vital for a smooth monitoring experience. API monitoring tools provide support in various ways, which include: * **Documentation**: Look for tools that offer well-written and detailed documentation. This should include clear instructions on how to set up and configure the tool, as well as troubleshooting guides for common issues. * **Community**: A strong and active community can be a valuable resource for users. Look for tools that have a vibrant community where users can ask questions, share experiences, and learn from each other. * **Customer support**: It's essential to have access to responsive customer support in case you encounter any issues or need assistance. Look for tools that offer multiple channels of communication, such as email or live chat, and have reasonable response times. * **Training and resources**: Some API monitoring tools provide additional training resources, such as webinars or tutorials, to help users understand the tool's features and capabilities better. By considering these parameters, you can confidently select an API monitoring tool that meets your specific requirements, enhances security, optimizes performance, integrates seamlessly, and provides reliable support throughout your monitoring journey. ## API Monitoring Best Practices Having a comprehensive API monitoring strategy is crucial for ensuring the smooth functioning of your APIs and identifying any issues before they cause major problems. To get the best from your API monitoring strategy, consider the following practices: | **Best Practice** | **Description** | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Set Clear Objectives | Clearly define your monitoring goals. Determine what aspects of your APIs you want to monitor, such as response times, error rates, or throughput. This will help you set up effective monitoring and focus on what matters most to your business. | | Establish Baseline Metrics | Before deploying your APIs, establish baseline metrics to understand their normal behavior. This will enable you to quickly identify any anomalies or deviations from the expected performance. | | Monitor Key Performance Indicators (KPIs) | Identify and monitor the key performance indicators that directly impact the user experience or business outcomes. This may include response time, error rates, latency, and availability. Tracking these KPIs will help you stay proactive in addressing potential issues. | | Implement Real-Time Alerts | Configure real-time alerts to promptly notify you when an API metric exceeds a predefined threshold or encounters an error. This allows for immediate action and minimizes downtime. | | Leverage Synthetic Monitoring | In addition to monitoring production APIs, consider implementing synthetic monitoring by simulating user interactions with your APIs. This enables you to proactively test and monitor critical functionalities without impacting real users. | | Analyze Historical Data | Analyzing historical monitoring data can provide valuable insights into trends, patterns, and potential areas for optimization. Use this data to make informed decisions and continuously improve your API performance. | By following these best practices, you can optimize your API monitoring strategy, minimize downtime, enhance performance, and ensure a seamless user experience. ## Conclusion In summary, API monitoring is essential for ensuring the security, performance, and reliability of APIs. It enables organizations to proactively detect and address issues, prevent security breaches, optimize performance, and deliver accurate data to users. By implementing effective monitoring practices and choosing the right tools, organizations can maintain the smooth functioning of their APIs. If you're considering implementing API monitoring in your projects, [try Checkly for free](https://app.checklyhq.com/signup). Explore our comprehensive monitoring solutions to enhance your API performance and ensure a seamless experience for your users. ## Frequently asked questions Learn more about API monitoring and how it helps you to avoid production issues. ### How often should you check the health of your APIs? There's no silver bullet when choosing the best monitoring interval. The main question is: How long can you afford broken API endpoints to go unnoticed? If you monitor your API endpoints only once an hour, and your API goes down, your discovery time can take up to one hour, too, depending on the time of the outage. We recommend choosing [short monitoring intervals for critical endpoints](https://www.checklyhq.com/blog/check-frequency/) going down to ten seconds to receive alerts almost in real time. For non-critical endpoints, a longer interval might suit you as well. ### Q: How can businesses ensure the correctness of API responses? When API responses change, this change often only appears when relying applications break. This results in a broken user experience. API monitoring prevents this from happening. A well-defined API monitoring solution enables developers to monitor API uptime but also API correctness and accuracy. [API check assertions](/docs/detect/synthetic-monitoring/api-checks/configuration) allow you to define what API responses you expect from your API endpoints. Check assertions range from testing essential HTTP response headers to well-formed and correct JSON responses. You know your API best and should be as specific as possible to avoid user-facing production issues. ### Q: How does 'Monitoring as Code' change traditional API monitoring approaches? Unlike traditional API monitoring services that rely on manual configuration, the [Monitoring as Code](https://www.checklyhq.com/product/monitoring-as-code/) approach enables developers to define and configure their API and end-to-end monitors in code. With Monitoring as Code, the entire monitoring configuration lives in source control next to the application code. The monitoring infrastructure can be tested in preview environments and will be updated when deploying to production via CI/CD. Having your monitoring setup defined in code eases the monitoring maintenance. Monitoring changes can be quickly retraced via version control; rollbacks can be applied via CI/CD and manual monitoring ClickOps will be banished. ### Q: What metrics are most critical when monitoring an API? The obvious [API monitoring metrics](https://www.checklyhq.com/learn/monitoring/metrics-every-team-needs/) are API uptime and correctness. An unavailable or broken endpoint directly affects user experience. But they're not the only metrics to monitor. API performance metrics are essential for a well-running API, too. Do your API endpoints respond to global HTTP requests in an acceptable time? And if they don't, are you aware of your API's degraded performance? To ensure a well-running API, you must constantly look at performance metrics because slow API responses will lead to a poor user experience of the products relying on these APIs. API response time is essential, but to find the cause of a slowed-down API, specific metrics such as DNS resolution time or time to first byte are handy tools to understand what's causing a performance regression. ### Q: How does API monitoring differ from website monitoring? Website monitoring describes practices to check a website's availability, correctness, and user experience. These checks can be performed in various ways, from [real user monitoring](/docs/learn/monitoring/real-user-monitoring/) to [synthetic monitoring](/docs/learn/monitoring/synthetic-monitoring/) mimicking real-user behavior with headless browsers. API monitoring takes an active approach to monitoring APIs. An API monitor performs scheduled HTTP requests and alerts you when API endpoints become unavailable or degrade in performance. ### Q: How do you handle false alarms in API monitoring? False alarms are a common problem in any active API monitoring solution because they'll trigger false alerts. API call retries are a standard solution to avoiding alerts caused by flaky APIs. A suitable retry solution is a balance between timely and valid alerts. How often do you want to retry until you get alerted? And how quickly do you want to retry an API call? And in what interval should your API be pinged again? The answers to these questions vary depending on the API and the urgency of downtime. A more extended retry period might be sufficient for some APIs, whereas, for others, a quick and single retry is the best approach. No matter your preference, your monitoring solution must provide [configurable retry strategies](/docs/communicate/alerts/overview#how-often-should-i-retry). # Frontend Monitoring - Benefits, Challenges, and Top Tools Source: https://www.checklyhq.com/docs/learn/monitoring/frontend-monitoring Discover the benefits, challenges, and top tools for frontend monitoring. Learn how to track performance, detect issues, and optimize user experience. No matter what internal testing or error monitoring we do for our web services, our end users will interact with that service through a front end. It’s necessary to perform front end monitoring so that you’re not relying on users to report problems. *** **Ready to start frontend monitoring today?** check out our practical guide on [how to start frontend monitoring with Checkly and Playwright.](https://www.checklyhq.com/guides/monitoring-ecommerce-apps-using-playwright/) *** Frontend monitoring ensures seamless user experiences by observing and analyzing the performance and functionality of web applications. Edge case failures, bad 3rd party service interactions, and poor front end performance are all examples of issues that only direct front end monitoring can detect. In this article, we’ll cover the basics of frontend monitoring, explore its various types, key metrics, benefits, and challenges, and review some top tools to help you manage and optimize your applications effectively. ## What is Frontend Monitoring? Frontend monitoring involves tracking and analyzing the performance, reliability, and user experience of web applications from the user’s perspective. Frontend monitoring can either occur by observing users real interactions on the frontend (real user monitoring) or by sending an automated system to interact with your frontend (synthetic user monitoring). It helps teams identify issues like slow page load times, JavaScript errors, or failed API calls that can negatively impact user satisfaction. ## Types of Frontend Monitoring **Proactive Monitoring** Proactive monitoring involves detecting potential issues before they affect users. Tools simulate user interactions to identify bottlenecks or vulnerabilities. For example, a proactive monitor might [simulate slow network response times](https://www.checklyhq.com/learn/playwright/intercept-requests/) to check how a frontend will render when some components are slow to load. This is comparable to a ‘chaos monkey’ testing strategy for backend services. **Reactive Monitoring** Reactive monitoring focuses on capturing issues reported by users in real-time. It complements proactive efforts by highlighting problems occurring in production. Reactive monitoring may be as simple as catching JS errors that occur in a page and reporting them to an end service. **Real User Monitoring (RUM)** RUM tracks actual user interactions with your application, offering insights into how real users experience your site in different environments. In theory, RUM is the ideal way to find front end problems: simply track every user’s experience, everywhere.

*A diagram of the Real User Monitoring model, with in-page Javascript reporting from a user's device to an observability service.* However this approach has several challenges: * Failures of users expectations may go undetected - for example if a user searches for recent posts and gets posts from 7 years ago, no errors will be raised, and no problem will be tracked. * Rum can impact browser performance for users. * Transmitting data for every user every time is quite expensive, both for you the service provider and for the user’s browser performance and network bandwidth. The suggested solution for this known issue is to sample randomly: when a user starts a session on your site, they’re randomly assigned whether that session will be tracked in detail and transmitted to your observability service. This raises the issue of missing key failures: when a key client reports an error on your site, but no data was captured, you’re stuck trying to replicate an issue with only a user description. * Difficulty finding patterns - user behavior is inherently inconsistent. It’s often very difficult to identify connected failures or trends based on multiple users’ inconsistent behavior. The situation is similar to the sampling problem: we’re left trying to guess what happened based on sketchy information. * Complex implementation - from loading a javascript package to track user experience in the browser, to endpoints to collect that information, and a system to find patterns in stochastic user behavior, RUM is a complex technical challenge with extensive technical lift. If you’re trying to create a DIY solution for RUM, you’ll find that [CORS: Cross-Origin Request Blocked](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors) errors are the first of many challenges. **Synthetic Monitoring** [Synthetic monitoring](https://www.checklyhq.com/blog/what-is-synthetic-monitoring/) can be as simple as sending requests to a service and making sure the response has a `200 OK` status code, but the term is generally used to describe an automated browser that can simulate user behavior. [Checkly uses the open source Playwright framework](/docs/detect/synthetic-monitoring/browser-checks/playwright-support) to simulate user behavior in great detail, and make complex assertions about the results. Synthetic monitoring solves many of the problems listed above with RUM: * Engineers can create frontend tests to be run that mimic user behaviors and key user paths. By scripting things like user searches, these tests can exactly describe the expected output. * Synthetic monitoring doesn’t impact performance for users. * Synthetic monitoring’s costs are controlled from the outset, as you control the cadence of synthetic test runs. * Patterns are readily identifiable since the behavior of a synthetic user is always the same. Further, by running tests on a cadence, the exact time that failures started is easier to find. This is especially helpful if you’re trying to connect a new failure to a particular deployment * No implementation requirements - synthetic monitoring can be implemented as a 100% external service.

*A diagram of the Synthetic Monitoring model, with an observability service like checkly sending requests to your production service. SREs and Ops engineers can then view reports and receives alerts the moment a problem is detected.* **Application Performance Monitoring (APM)** APM is a general term, usually referring to a combined solution that includes both front end monitoring and the measurement of performance of backend software through the use of installed software agents. *** ## How Frontend Monitoring Works Frontend monitoring tools capture data through a combination of synthetic tests, browser instrumentation, and real user interactions. These tools generate insights by analyzing performance metrics, logging errors, and tracking user behavior. *** ## Key Components of Frontend Application Monitoring * **Performance Metrics**: Monitor loading speeds, [rendering times, and resource usage](https://www.checklyhq.com/learn/playwright/performance/#performance-with-headless-tools). * **Error Detection**: Identify JavaScript errors, API failures, and crashes. Crash reporting can be a complex problem but it’s great if it’s possible to report some details from user browser crashes. * **User Experience Analysis**: Assess user interactions, engagement, and satisfaction. This general concept is only sensible when performing Real User Monitoring, and may have significant overlap with business intelligence or business analytics. If you find yourself asking ‘what interface elements are most attractive to users?’ your use of the tool has shifted from monitoring to user analytics. *** ## Key Metrics in Frontend Monitoring The key metrics for frontend monitoring—*performance*, *user interactions*, *errors*, and *availability*—are crucial for understanding and optimizing the user experience. **Performance** metrics, including page load time and Core Web Vitals (e.g., Largest Contentful Paint, Interaction to Next Paint), measure how fast and smoothly content is delivered to users. **User interactions** capture events like clicks, form submissions, and navigations to gauge how users engage with the application. **Errors** track JavaScript exceptions, resource failures, and API issues, helping teams identify and resolve defects impacting functionality. **Availability** monitors uptime and service reachability to ensure the application is consistently accessible. Together, these metrics provide a comprehensive view of application health, enabling teams to improve performance, address issues proactively, and enhance the overall user experience. Key metrics in detail: * **Core Web Vitals**: [Largest Contentful Paint](https://web.dev/articles/lcp) (LCP), [Interaction to Next Pain](https://web.dev/articles/inp) (INP), Cumulative Layout Shift (CLS). These metrics are considered to be quite critical for search engine optimization. * **JavaScript Error Rates**: Frequency of client-side code failures. You may need to implement filtering for common errors. * **API Response Times**: Speed and reliability of API calls. This may be called ‘[heartbeat monitoring](/docs/detect/uptime-monitoring/heartbeat-monitors/overview)’ if you’re only measuring the reliability of straightforward `get` requests. * **Network Request Failures**: Broken or delayed network requests * **Errors and Crashes**: Stability of the application under different conditions * **User Interactions and Engagement**: Clicks, scrolls, and session durations *** ## Common Use Cases of Frontend Monitoring * **Monitoring Page Load Times**: [Ensure optimal page rendering speed](/docs/detect/synthetic-monitoring/browser-checks/performance-metrics#total-blocking-time). * **Tracking Third-Party Services**: No amount of pre-deploy testing with stubs of third party services can find all the possible interaction problems with those services. Frontend monitoring can detect issues caused by external libraries or APIs. * **User Interaction Monitoring**: Analyze behavior patterns and engagement levels. * **Analyzing Client-Side Errors**: Identify and resolve JavaScript issues, especially those happening on single platforms or with particular browser versions. *** ## Benefits of Frontend Monitoring **Proactive Issue Detection** Frontend monitoring empowers teams to detect potential problems before they escalate into critical failures. By observing patterns such as sudden increases in error rates or performance degradation, teams can investigate and resolve issues early, often before users are impacted. This phase of the monitoring journey focuses on "known knowns," enabling developers to answer key questions like, “What happened during a spike in errors?” By building a narrative around historical data, teams can use these insights to improve system resilience. **Precise Performance Insights** [Performance monitoring](/docs/detect/synthetic-monitoring/browser-checks/performance-metrics) provides actionable data that highlights bottlenecks and inefficiencies in applications. Metrics such as Largest Contentful Paint or Interaction to Next Paint help developers understand where delays occur and prioritize optimization efforts. These insights shift the focus from merely reacting to known issues toward analyzing anomalies, such as unexpectedly fast or slow responses. This phase aligns with "known unknowns," where developers explore statistical questions to assess how normal or abnormal system behavior is. **Real-Time User Experience Analysis** Frontend monitoring enables developers to track user interactions, such as clicks, form submissions, or navigation paths, providing visibility into how users engage with an application. These insights help identify friction points in real-time, allowing teams to address usability challenges proactively. By adding context-specific data, such as user or session identifiers, developers can refine their analysis and enhance the customer experience. This phase blends analysis with experimentation, helping developers test hypotheses about what improves engagement or usability. **Resource Allocation Efficiency** By identifying specific areas that require attention, monitoring data enables teams to focus their efforts where they matter most. For instance, if metrics reveal that a checkout flow is a primary source of performance complaints, resources can be redirected toward optimizing that functionality. This approach not only improves the end-user experience but also ensures the efficient use of time and budget. This aligns with the later phases of the observability journey, where teams experiment with targeted solutions and evaluate the ROI of their changes. *** ## Challenges of Frontend Monitoring **Managing Tool Sprawl and Integration Issues** The abundance of frontend monitoring tools can create silos, making it challenging to consolidate insights across systems. Teams often struggle to integrate disparate tools like performance trackers, error loggers, and user interaction analytics. This fragmentation can hinder the ability to see the full picture, leaving blind spots in observability. Addressing this challenge requires adopting systems that unify logs, metrics, and traces to provide a cohesive view of the application’s behavior. **Handling Diverse User Environments** Applications are accessed through a wide array of devices, browsers, and network conditions, each introducing unique challenges. For example, a feature that works seamlessly on one browser may fail on another, and network latency can vary dramatically across regions. Monitoring tools must account for this diversity by capturing data that reflects real-world conditions. Understanding these "known unknowns" helps developers identify anomalies across different environments and adapt their solutions to improve the experience for all users. **Keeping Up with Rapid Technological Changes** The fast pace of evolution in frontend technologies, including frameworks and browser standards, makes it difficult to maintain effective monitoring. What works for today’s stack might become obsolete as new features and tools emerge. Teams must stay agile, continually updating their instrumentation and observability practices to align with the latest advancements. This requires a culture of experimentation and learning, allowing developers to test and adopt new solutions without losing focus on system stability. **Limited Root Cause Analysis with Basic Tools** Entry-level monitoring tools often focus on surface-level insights, such as error counts or simple performance metrics, but fail to provide the deeper diagnostics needed to identify root causes. For example, they may show that an API call failed but not explain why. This limitation makes it difficult to move from the "what" to the "why," hindering the ability to address systemic issues. Advanced observability tools that provide context-rich data and enable correlation across systems are essential for overcoming this challenge. **Ensuring Full-Stack Visibility** Frontend monitoring alone cannot provide a complete view of an application’s health. Many issues arise at the intersection of the frontend and backend, requiring integrated observability across the entire stack. Without this integration, teams risk spending excessive time proving whether an issue is frontend- or backend-related. Implementing [context propagation](https://www.checklyhq.com/learn/opentelemetry/context-propagation/) and unified tracing, such as through OpenTelemetry, helps connect frontend events to backend processes, enabling a more holistic understanding of system behavior and streamlining troubleshooting efforts. For more detail on connecting backend [OpenTelemetry traces](https://www.checklyhq.com/learn/opentelemetry/traces/) with frontend performance see how [Checkly Traces connects data from across your stack](/docs/resolve/traces/how-it-works). ## Top frontend monitoring tools | Tool | Features | Notes | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Checkly** | Uses Playwright [end-to-end tests for synthetic monitoring](/docs/detect/synthetic-monitoring/overview). Robust alerts to detect issues early. Integrates with OpenTelemetry traces. | Excellent for proactive monitoring and integration with backend systems. | | **Sematext** | Backend monitoring with [an open-source data collection agent](https://github.com/sematext/sematext-agent-java). | Suitable for teams needing a lightweight, open-source-friendly solution. | | **Pingdom** | Specializes in uptime monitoring and performance tracking. | Focused on simple up-or-down monitoring. Limited in scope compared to other tools. | | **Google PageSpeed Insights** | Provides performance recommendations based on real-world data. | Primarily an auditing tool, best for reactive monitoring. | | **New Relic Browser** | Offers frontend monitoring with APM integration. Provides Real User Monitoring (RUM) capabilities. | Comprehensive but costly, best for large-scale applications needing detailed user insights, willing to be ‘locked in’ to the closed New Relic ecosystem | | **Sentry** | Focused on error tracking more than traditional monitoring. | Ideal for teams prioritizing debugging JavaScript errors and crashes. | | **Dynatrace** | Full-stack monitoring with AI-driven insights. | Expensive, with limited innovative features. AI insights often summarize existing data. | | **AppDynamics** | Monitors both frontend and backend performance, another ‘all in one’ APM tool. | Costly enterprise-level tool, suited for large organizations with complex environments. If you’re using AppD, chances are your team has been using it for 5+ years! | *** ## How to Choose the Right Frontend Monitoring Tool Consider features like synthetic monitoring, RUM, session replay, language compatibility, and security. Evaluate pricing, ease of use, and whether an agent or agentless setup fits your needs. ## Frontend monitoring and OpenTelemetry Frontend monitoring has historically lagged behind backend systems in sophistication and integration. With OpenTelemetry (OTel), frontend monitoring can now achieve deeper insights into user experiences. However, it also brings unique challenges that require careful consideration. ## Challenges in Using OpenTelemetry for Frontend Monitoring\*\* **1. Initial Complexity of Setup** While OpenTelemetry provides out-of-the-box instrumentation, setting it up for a browser-based application requires upfront effort. Instrumentation code must load before the application initializes to capture critical spans like document load times. Ensuring this is correctly implemented across environments can be a source of frustration. **2. Performance Overhead** Instrumenting a frontend app involves adding listeners for browser events, fetching metrics, and propagating trace headers. Over-instrumentation or poorly optimized spans can degrade application performance, especially for resource-intensive pages or on devices with limited capabilities. **3. Handling Browser-Specific Nuances** Browsers have unique behaviors that can make instrumentation challenging. For example: * **Redirects and Network Timing:** JavaScript doesn’t have access to certain browser-level events, like pre-redirect network timing. Combining OpenTelemetry's network instrumentation with browser APIs like `PerformanceObserver` can help, but it requires extra configuration. * **Clock Synchronization:** Distributed tracing relies on accurate timestamps. However, clock drift between client devices and servers can result in misaligned spans. Proxying timestamp corrections through an [OpenTelemetry Collector](https://www.checklyhq.com/learn/opentelemetry/what-is-the-otel-collector/) is often necessary. **4. Data Volume and Rate Limiting** Frontend telemetry can quickly generate a high volume of spans, especially on high-traffic applications. Without rate limiting or filtering, this data can overwhelm storage systems or increase monitoring costs. Developers must design selective instrumentation strategies to focus on high-value spans. **5. Contextual Relevance of Spans** While auto-instrumentation provides useful baselines like resource fetch times and click events, it lacks application-specific context. Developers need to enhance these spans with attributes that matter to their business logic—such as user IDs, session data, or interaction details. Without this, telemetry risks becoming just noise. **6. Debugging and Observability Gaps** Traditional frontend monitoring tools often report *what* is happening (e.g., a page is slow), but lack insights into *why*. OpenTelemetry addresses this by correlating frontend spans with backend traces. However, this requires propagation of trace headers (`traceparent`) between the frontend and backend, which can be technically challenging in distributed systems. OpenTelemetry opens a new frontier for frontend observability, but it’s not a plug-and-play solution. Teams must balance instrumentation depth with performance and focus on collecting the data that best illuminates user experiences. By addressing these challenges head-on, developers can achieve a robust, connected view of their systems and elevate the standard of frontend monitoring. ## Developing a Frontend Monitoring Strategy: A Step-by-Step Guide 1. **Identify Key Metrics**: Focus on what matters most to your application. 2. **Select Tools**: Choose tools that integrate with your tech stack. 3. **Monitor During High-Usage Periods**: Analyze performance under peak load. 4. **Integrate Insights**: Align findings with business goals. *** ## How Can Checkly Help with Frontend Monitoring? Checkly performs synthetic monitoring with the power of Playwright for robust testing of key workflows. By simulating real-world scenarios, it ensures applications remain functional and performant under varying conditions. Effective frontend monitoring ensures the seamless performance and reliability of websites and applications, delivering a strong user experience. Checkly excels in this area by combining the power of end-to-end (E2E) monitoring and modern development workflows to proactively identify and resolve issues before they impact users. Checkly enables developers to monitor critical user flows—like login, search, and checkout—in real-time. By leveraging tools like Playwright, Checkly runs automated scripts simulating real user interactions. This synthetic monitoring approach provides continuous feedback on the health of frontend applications, revealing issues that often go undetected in pre-production testing. For example, Checkly allows teams to: * Run E2E tests on a cadence to catch edge cases and unpredictable failures. * Test frontend flows directly in production, monitoring the behavior users actually experience. * Extend monitoring to include application performance metrics like load times and rendering speed. **Advantages of Checkly’s Approach** 1. **Integration with Existing Workflows** Checkly fits seamlessly into CI/CD pipelines, enabling checks to trigger automatically on deployments or pull requests. This "[monitoring as code](https://www.checklyhq.com/guides/monitoring-as-code/)" (MaC) approach allows developers to maintain monitoring scripts alongside the application codebase. 2. **Resource-Efficient Testing** By utilizing headless browser automation tools, Checkly achieves higher stability and faster execution compared to traditional headful testing methods. This means frontend monitoring can run efficiently in cloud environments without consuming excessive resources. 3. **Real-Time Alerting** When checks fail, Checkly alerts teams immediately through preferred channels like Slack, PagerDuty, or email. This ensures swift action to resolve issues and minimize user impact. 4. **Global Monitoring Capability** Checkly's cloud-based infrastructure lets teams run tests from multiple geographic locations, ensuring frontend performance remains consistent for users worldwide. ## Conclusion Frontend monitoring is essential for maintaining application reliability and delivering exceptional user experiences. By leveraging the right tools and strategies, teams can proactively address issues, optimize performance, and meet the expectations of modern web users. Consider tools like Checkly to simplify and strengthen your monitoring efforts. # HTTPS Fundamentals - Beginner's Guide for App Developers Source: https://www.checklyhq.com/docs/learn/monitoring/https-for-app-developers Let’s break down what HTTPS is, why it matters, and how to use it in your apps. **HTTPS Fundamentals for Application Developers** When you build apps, keeping user data safe is key. One way to do this is by using HTTPS. Let’s break down what HTTPS is, why it matters, and how to use it in your apps. *** ### What is HTTPS? HTTPS stands for **HyperText Transfer Protocol Secure**. It’s the secure version of HTTP, the protocol used to send data between a user’s browser and your app’s server. The “S” in HTTPS means the data is encrypted, so no one can read or change it while it’s being sent. It’s not to be confused with the rarely used Secure HTTP (S-HTTP). *** ### Why Use HTTPS? 1. **Protects Data**: HTTPS encrypts data like passwords, credit card numbers, and messages. HTTPS is crucial on insecure networks (like public Wi-Fi) to prevent data tampering or theft. I 2. **Builds Trust**: Users see a padlock icon in their browser when your app uses HTTPS. This tells them your app is safe. 3. **Improves SEO**: Search engines like Google rank HTTPS websites higher. 4. **Improves Compatibility**: Users on many browsers and devices will receive warnings when not using HTTPS. *** ### How HTTPS Works HTTPS uses two main tools to secure data: 1. **SSL/TLS Certificates**: These are digital certificates that prove your app’s identity. They also help encrypt the data. 2. **Encryption**: HTTPS scrambles data into a code that only the server and browser can unscramble. HTTPS uses the same syntax as HTTP but adds a layer of encryption through transport layer security (or formerly secure sockets layer, hence SSL/TLS) to secure data. This encryption protects against eavesdropping and man-in-the-middle attacks, even if only the server is authenticated (via its certificate). HTTPS encrypts the entire HTTP protocol, including URLs, headers, and cookies, but it cannot hide the server’s IP address, port, or domain name. Browsers trust HTTPS websites based on pre-installed certificates from trusted Certificate Authorities (CAs). For a secure connection, users must trust their device, browser, the CA, and the website’s valid certificate. t also supports modern protocols like HTTP/2 and HTTP/3 for faster performance. To enhance security, developers should use HTTP Strict Transport Security (HSTS) to prevent SSL stripping attacks. HTTPS is essential for protecting user privacy, especially as global surveillance and data theft increase. It’s not to be confused with the rarely used Secure HTTP (S-HTTP). *** ### How to Add HTTPS to Your App 1. **Get an SSL/TLS Certificate**: * You can buy one from a Certificate Authority (CA) or get a free one from services like Let’s Encrypt. * Install the certificate on your server. 2. **Update Your Server**: * Configure your server to use HTTPS. * Redirect all HTTP traffic to HTTPS so users always use the secure version. 3. **Test Your Setup**: * Use tools like SSL Labs ([https://www.ssllabs.com/](https://www.ssllabs.com/)) to check if your HTTPS setup is correct. * Make sure your app works smoothly with HTTPS. * Monitor your connections with an automated monitor like [Checkly](/docs/concepts/results). *** ### Common Mistakes to Avoid * **Mixed Content**: Don’t mix HTTP and HTTPS. For example, if your app uses HTTPS but loads images over HTTP, it can break the security. * **Expired Certificates**: Always renew your certificates on time. * **Weak Encryption**: Use strong encryption methods to keep data safe. * **Certificate Expiry:** Renew your SSL/TLS certificate before it expires. *** ### Conclusion HTTPS is a must for modern apps. It keeps user data safe, builds trust, and helps your app rank better. By following the steps above, you can easily add HTTPS to your app and avoid common pitfalls. Start using HTTPS today—it’s one of the best ways to protect your users and your app. Happy coding! 🚀 # Understanding the Software Development Life Cycle (SDLC) for Engineers Source: https://www.checklyhq.com/docs/learn/monitoring/intro-to-sdlc With fast-paced standards for development and operations; where Agile, DevOps, and continuous delivery dominate—understanding SDLC remains crucial. The **Software Development Life Cycle (SDLC)** is a structured process that guides the creation, deployment, and maintenance of software systems. It provides a framework for engineering teams to efficiently design, develop, test, and release high-quality software while minimizing risks and costs. With fast-paced standards for development and operations; where Agile, DevOps, and continuous delivery dominate—understanding SDLC remains crucial. However, traditional models are evolving, and testing strategies (like the Testing Pyramid) are being re-evaluated. ## **Phases of the SDLC** The SDLC typically consists of several key phases: 1. **Planning & Requirements Analysis** * Define project scope, objectives, and stakeholder needs. * Gather functional and non-functional requirements. 2. **System Design** * Architectural decisions, tech stack selection, and system specifications. * Design and documentation of requirements * Divide the project up into tasks and sprints. 3. **Implementation (Coding)** * Development of features based on design specifications. * Track individual tasks and sprints to see if development is on target for planned release date. 4. **Testing** * Ensures software meets requirements and is free of defects. * Includes unit, integration, system, and acceptance testing. * The **Testing Pyramid** (unit > integration > UI tests) has been debated—some argue it doesn’t fit modern microservices and fast deployments. 5. **Deployment** * Release to production, often via automated pipelines in DevOps environments. 6. **Maintenance & Operations** * Bug fixes, performance improvements, and feature updates. * **Monitoring and observability** are critical in this phase. The Testing Pyramid

*The traditional testing pyramid, inspired by Martin Fowler’s version* ## **The Changing Landscape of Testing in the SDLC** The traditional **Testing Pyramid** (favoring many unit tests over fewer UI tests) as a support of the SDLC is being questioned. Challenges include: * **Surprise Interactions**: with so many microservices and third-party services, it’s very difficult to know from unit tests or even integration tests that the code written locally will perform correctly for users. * **Microservices & Distributed Systems**: Integration and contract testing gain importance. * **Shift-Left Testing**: Testing earlier in the cycle to catch defects sooner. * **Test Automation & AI**: Tools like AI-driven test generation are changing how tests are written and maintained. Currently, there’s a great deal of conflict as [AI tools struggle to make tests as useful](https://www.youtube.com/watch?v=XI8Q_Yx1vqk) as human writers. As technologies evolve, it may be a better choice to use the same tests all the way [from the development phase up to production monitoring](https://www.checklyhq.com/guides/sdlc-monitoring/). ## **Conclusion** The SDLC remains foundational for engineering teams, but its execution must adapt to modern practices. Testing strategies, in particular, must evolve with architectural shifts. By understanding SDLC principles and staying flexible in testing approaches, engineers can deliver robust, scalable software efficiently. # Observability Basics - Metrics that every team needs Source: https://www.checklyhq.com/docs/learn/monitoring/metrics-every-team-needs This article will help define the metrics that every team needs to get tracking of their system’s performance When you begin implementing observability beyond basic logging, you’ll be presented with a large number of tools that promise the ability to give insight into every aspect of your application. But when we consider observing every possible running line of code and service worker stats, we quickly find ourselves going from almost no information to information overload. Rather than trying to observe everything, it’s better to start with a minimum viable product, and this article will help define the metrics that every team needs to get tracking of their system’s performance. Along with describing these key metrics, we’ll also discuss the design questions you should ask early on when defining these metrics. The goal is to create metrics that are useful in two ways: * These metrics should offer an immediate picture of how your system is performing * These metrics should be stable enough that you can do historical comparisons and see a change in performance The four key metrics for monitoring your application are all connected to each other, and without any of them you will miss the cause of some failures. ## Throughput Throughput is an APM (Application Performance Monitoring) metric that measures how much work a system can handle in a given time. It counts the number of requests, transactions, or tasks completed successfully. For example, in a web app, it could track how many user requests the server processes per second. Higher throughput means the system is handling more work efficiently. Probably the easiest metric to measure, it gives a quick view of how much capacity is in use. This author started out in the tech sector doing phone tech support, and one evening I had a report that our core service was totally unavailable. When checking the main site it loaded for me, but when I used a network location outside our office, sure enough the site didn’t load. When working with the on-call engineer, they checked the server’s available memory and response time, and everything looked great. The response was obvious once we thought for a moment: the system was performing extremely well for the handful of users who could get access, everyone else was being blocked by network issues. It’s a story I have often recounted, but I tell it here again because it highlights an important point: **without the rate of traffic for your service, all other performance metrics are meaningless**. ### Best practices for measuring throughput While there’s generally little disagreement about the definition of throughput, care should be taken that services that handle requests asynchronously, or those that subdivide a user’s request, don’t count multiple ‘requests’ for a single user request. Special throughput numbers can be created for services like data stores, and these may have higher throughput numbers than the total number of user requests, however in general we should equate throughput with ‘the number of user requests.’ ## Response time Response time measures how fast a system or application responds to a user’s request. It tracks the total time from when a user clicks a button, searches, or loads a page until they see the result. Faster response times mean a smoother experience. Slow response times can frustrate users and hurt performance. Keeping response times low is key for a good user experience. Monitoring response times over time can help show whether the product team as a whole is effectively adding features while maintaining performance. For browser based applications, a long response time may motivate users to go elsewhere; in IoT devices, long response times may make users think their appliance is broken! ### Best practices for measuring response times Unlike throughput, response time is a metric with multiple possible interpretations. If a user is running our UI on an extremely old and slow device, and despite returning image data after 4.2 seconds, it takes 12 seconds for the image to render, what was the response time? The question of ‘does network time, frontend rendering time, etc count as part of our response time’ doesn’t have a right answer, it’s just a question of deciding in advance what you’re measuring and sticking with it.

*The layers involved in a response can cause some language confusion.* For some services there won’t be much confusion. For example an API service should just be measured on how long it takes to reply to a query. Of course, even with an API, dependencies like databases and message queues will have their own response times, which should be clearly labelled. In general, the best practice is to **clearly label any response time that isn’t covering end-to-end response time**, namely ‘how long it takes from a user clicking a button to when they see the result they’re expecting,’ as being a special case versus the standard expectation for response time. Of the metrics in this article, response time can be the most difficult to measure accurately. While you can see request and response times within your own cloud, it’s quite difficult to see whatg users are actually experiencing with their own browsers and network connections. To measure response times, rather than trying to observe every chunk of time spent in your backend service, **consider Checkly to actively monitor your service and record how long each request really takes** for common user pathways. Finally, when viewing response times **it’s critical not to just use simple averages to capture performance**. Special statistical slices like ‘percent of responses over threshold’ (how often did a user get an unacceptably slow response) and ‘95th percentile response times’ (including all but the very slowest responses, which are often outliers caused by special requests that are predictably quite slow) is critical to get an accurate picture of performance over time. ## Infrastructure Metrics You should always be able to see a small group of metrics that shows the health of your infrastructure. This article doesn’t give a single answer for these infrastructure metrics due to evolution of how we host applications, and what things stress us out now. In the old days of bare metal servers, infrastructure health was usually covered by looking at available memory and CPU usage. But if a team using mainly serverless computation on a public cloud, those numbers would be both meaningless and ephemeral, and of bigger concern is a metric showing how much is being spent to handle all customer requests. Some possible examples of key infrastructure metrics: * Infrastructure usage/estimated spend - running on a smoothly scaling public cloud, we’re probably not concerned that AWS will run out of RAM, but rather we care if changes to our application have caused us to spend twice as much to handle the same number of user requests. * Container health - on a managed cluster, questions like ‘how many containers are stuck in a crash loop?’ or ‘what percentage of time are all services available?’ probably encapsulate most of what we care about. * Unused infrastructure - if, rather than a serverless or other auto-scaling environment, our cloud usage is pre-negotiated and remains stable even during low traffic, it may be that our biggest concern is when we’re paying to run capacity that isn’t actually being used. These metrics are still critical for any operations or on-call team, and are especially critical when things don’t add up, for example when throughput is way down, but response times are also increasing. ### Best practices for monitoring infrastructure The best practices for infrastructure management could fill a whole book, but at the very highest level the most critical skill for monitoring infrastructure is to **perform deep, blameless postmortems any time you have an outage that was user-reported**. Outages reported by users are failures that your current observability tools can’t detect. By asking ‘what did we miss’ you’ll begin to identify what needs to be measured better, and surfaced better for your engineers, from your infrastructure. ## Error Rate Error rate is a critical APM metric that quantifies the frequency of failed requests or transactions within an application, expressed as a percentage of total requests. It captures HTTP errors (e.g., 4xx client errors, 5xx server errors), timeouts, exceptions, and can cover business logic failures (e.g., payment declines, API call failures). This metric is often segmented by endpoint, service, or dependency to pinpoint failure hotspots. A high or rising error rate may indicate code defects, infrastructure instability (e.g., database overload), or third-party service degradation. Engineering teams use error rate alongside logs and distributed tracing to diagnose root causes. In SLOs (Service Level Objectives), error rate thresholds (e.g., \<0.1%) help enforce reliability targets. Proactive monitoring and alerting on error rate spikes enable faster [incident response](https://www.checklyhq.com/learn/incidents/what-is-incident-response/), reducing MTTR (Mean Time to Repair) and minimizing user impact. One of the things that makes a metric a key metric is that without observing it, you can have your application fail silently. For example if our frontend is rendering our site beautifully, but the site content is an error message, the response time may look fine, throughput may be fine (or even up since users are hitting reload repeatedly) and our infrastructure metrics may not show a problem. But clearly, this error message should be spiking our application’s error rate. ### Best practices for measuring error rate Error rate will have by far the most disagreement in your team about what counts as an error. In the section above we included things like declined card payments as errors, though that may bristle for some developers, since presumably all our code and our UI can handle a declined card payment just fine, and prompt the user to try a different card. There is no single best answer for what counts as an error, but you should answer the following questions before you start measuring an error rate: * Do errors raised by code that are not observed by users count as errors? * Can a single request result in more than one error? * Do business logic failures count as errors (for example incorrect passwords or API keys, payment declines, or error responses from third party services)? However you answer these questions, one other best practice is to **always count errors that surface to the user**, even if no error code is raised. One rather old example is when the team implements nicesly rendered pages to the ll the user that they’ve encountered an invalid URL or that their request returned a database error. These incidents should all be counted as errors, since the more users see these messages, the more they’ll perceive that something is wrong with this application. For example the following API response: ```json theme={null} HTTP/1.1 200 OK Server: nginx/1.18.0 Date: Wed, 03 Apr 2024 12:00:00 GMT Content-Type: application/json; charset=utf-8 Content-Length: 64 Connection: keep-alive Access-Control-Allow-Origin: * { "body": "Error code 500 an internal server error has occured" } ``` This kind of response, with a `200 OK` status and an error in the body, may not be great API design, but it should still be registered in your system as an error. ## Conclusions: the sturdy table of your application For all of these four key metrics, I can think of examples where only one of them will be ‘off’ during a certain type of failure. Therefore all four are the table stakes to monitoring a production application. ## **Conclusions: The Sturdy Table of Your Application** Effective observability begins with tracking **four foundational metrics**: **Throughput**, **Response Time**, **Infrastructure Health**, and **Error Rate**. Each of these metrics provides a unique lens into system performance, and missing any one of them can leave critical blind spots in your monitoring strategy. * **Throughput** tells you *how much* your system is handling. * **Response Time** reveals *how fast* it’s serving users. * **Infrastructure Metrics** ensure the underlying systems are healthy and cost-efficient. * **Error Rate** exposes *how often* things are failing—even when failures are silent. ### **Key Takeaways** 1. **No Single Metric Tells the Whole Story** 2. **Define Metrics Clearly and Consistently** * Avoid ambiguity in what constitutes a "response time" or an "error." * Document whether metrics include network latency, frontend rendering, or just backend processing. 3. **Use Percentiles, Not Just Averages** * Track **P95/P99 response times** to catch latency outliers. * Monitor **error rate trends** (not just spikes) to detect gradual degradations. 4. **Adapt Infrastructure Metrics to Your Stack** * Serverless? Focus on cost efficiency and cold starts. * Kubernetes? Track container restarts and pod availability. * Bare metal? Watch CPU, memory, and disk I/O. 5. **Error Rate Should Reflect User Experience** * Even "soft" errors (like payment declines) should be tracked if they impact UX. * Beware of **false-success responses** (HTTP 200s with error messages). ### **Final Thought: Start Small, Then Expand** While advanced observability (heatmaps, business intelligence, etc.) is valuable, these four metrics form the minimum viable dashboard every team should monitor. Once they’re stable, you can layer on deeper insights—but without them, you’re flying blind. By measuring these fundamentals, your team can: ✔ Detect anomalies faster ✔ Diagnose root causes more accurately ✔ Prevent minor issues from becoming major outages Part of starting small is considering the most basic questions first: with Checkly, you can create an automated service to simulate common user requests, and report immediately if anything looks incorrect. Checkly’s [synthetic monitoring](/docs/learn/monitoring/transaction-monitoring) ensures that you’re not relying on users to report problems. *** **Next Steps:** * Set up [dashboards for these four metrics](http://localhost:3000/learn/opentelemetry/what-is-the-otel-collector/). * Define your [alert thresholds](https://www.checklyhq.com/docs/communicate/alerts/configuration). * Define your [playbooks for incident response](http://localhost:3000/learn/incidents/playbooks/). * [Add Checkly](/docs/detect/synthetic-monitoring/browser-checks/overview) to find out about outages before your users do # Monitoring as Code - A Practical Guide from Checkly Source: https://www.checklyhq.com/docs/learn/monitoring/monitoring-as-code Learn how to implement monitoring as code with Checkly's guide. Bring transparency, scalability, and efficiency to your cloud infrastructure management. [Synthetic monitoring](/docs/learn/monitoring/synthetic-monitoring/) is a proven practice for ensuring reliability of applications and services in production. It excels at proactively discovering issues that affect real users, which is why companies—from startups to Fortune 500—rely on it to enhance their user experience. But teams that are moving fast or growing at scale can face real challenges with traditional synthetic monitoring workflows and tools: * Noisy alerts * Limited visibility into root causes * Siloed tools * Expensive and time-consuming scaling of monitoring Monitoring as Code (MaC) addresses these pain points by automating and integrating monitoring directly into CI/CD pipelines, delivering efficiency, scalability, and actionable insights. It redefines monitoring as a dynamic, code-driven process that evolves seamlessly with your applications. Let’s explore MaC, its benefits, and how you can use it to level up your monitoring. ## What Is Monitoring as Code? Monitoring as Code involves defining monitoring configurations and operations as code, enabling automated setups, real-time observability across DevOps lifecycles, and seamless integration with development workflows. With Monitoring as Code, your monitoring is implemented in your CI/CD lifecycle and maintained in version control, without having to rummage around a UI. This approach ensures that monitoring evolves alongside your application, providing continuous feedback and maintaining stability. ## Challenges of traditional synthetic monitoring While synthetic monitoring is a tried and true software practice, there are a few challenges that come with the way tools have traditionally worked. ### Noisy, fatiguing alerts Traditional monitoring tools are difficult to configure alerts at a granular level, often over alerting a chain of teams, wasting time and precious resources. This leads to “alert fatigue,” where important alerts can be missed, and teams become desensitized to notifications. ### Hard to scale in large systems Synthetic monitors can be difficult to implement and maintain in large, complex systems. Organizations with many services and screens can spend hundreds of hours creating and reconfiguring monitors and alerts as their underlying software changes. ### Siloed tools, metrics, and alerts Operations teams have traditionally owned monitoring tools within organizations and often times are siloed from the engineering teams who ultimately need to fix the error. ### Limited visibility into distributed systems Traditional synthetic monitoring tools work at the surface level of an application, unable to give detailed explanations of the root cause of the alert. ### Growing Costs Whenever we discuss the ideal setup for synthetic monitoring, experienced DevOps and SRE engineers will respond: "Sure, that would be great, if we had the budget for it." While it would be nice to simulate a complex user behavior every 30 seconds from 5 different geographic regions, the cost of most synthetic tools makes that cost-prohibitive. ## Benefits of Monitoring as Code Here are the most important benefits of adopting Monitoring as Code: ### **Codified & Programmable** Monitoring as Code transforms your monitoring setup into code that is stored, managed, and versioned alongside your application code. This approach brings significant advantages over traditional monitoring tools: * **Advanced Programmability**: The programmable nature of MaC allows you to achieve levels of automation and customization that are impossible in traditional UI-based tools. You can write scripts, create reusable templates, and use wrappers to dynamically generate and maintain monitors based on your application code. This approach simplifies the management of complex systems and ensures monitoring scales effortlessly with your application. * **Integrated with CI/CD Pipelines**: Monitoring configurations are seamlessly integrated into your CI/CD processes, deploying checks automatically as part of your application updates. This eliminates manual setup, ensures comprehensive coverage, and reduces the risk of human error. * **Version Controlled**: Every change to your monitoring configuration is tracked, enabling clear audits and easy rollbacks when needed. This ensures accountability and simplifies compliance, especially in highly regulated industries. With MaC, your monitoring evolves alongside your codebase, enabling precision, flexibility, and scalability that traditional methods simply can’t match. VS Code showing the checkly.config and a define API monitor.

VS Code showing the checkly.config and a define API monitor.

### **Reusability Enforces Consistency** Reusability in Monitoring as Code (MaC) goes beyond efficiency—it ensures consistent, reliable, and scalable monitoring across environments and teams. By reusing existing assets like Playwright tests and API definitions, MaC streamlines setup and eliminates redundancy. * **Leverage Existing Code**: Use Playwright tests and API definitions you’ve already written to create monitors. This eliminates the need to duplicate effort and ensures your monitoring is always in sync with your application’s logic and behavior. * **Simplified Scaling**: Reusable configurations allow you to replicate monitoring setups across multiple environments or projects without starting from scratch. This is especially valuable when scaling to new services or regions. * **Faster Onboarding and Setup**: Pre-tested modules and definitions reduce setup time for new services and simplify onboarding for engineers, allowing teams to deploy consistent, best-practice monitoring quickly. * **Standardization Across Teams**: Shared monitoring assets enforce best practices, ensuring all teams operate with the same standards and reducing discrepancies between environments. By reusing existing assets and creating modular configurations, MaC ensures monitoring is efficient, consistent, and adaptable to changing needs. ### **Increased Collaboration** MaC breaks down silos by embedding monitoring into the development lifecycle, fostering collaboration across teams: * **Unified Workflows**: Developers and Ops teams collaborate on monitoring in the same repositories they use for application code. Everyone speaks the same language—code. * **Cross-Team Visibility**: Pull requests and reviews for monitoring setups promote transparency, enabling teams to align monitoring with application behavior and expectations. * **Shared Ownership**: Monitoring shifts from being a secondary concern to a shared responsibility. Developers, SREs, and DevOps engineers work together to ensure robust monitoring for applications. By embedding collaboration into the monitoring process, MaC drives alignment, reduces friction, and creates a culture of shared accountability. ## Monitoring as Code Use Cases ### **Continuous Monitoring Throughout the SDLC** To maintain high-quality deployments, monitoring should span every stage of the Software Development Life Cycle (SDLC)—from local development to staging and production. Monitoring as Code (MaC) integrates directly into CI/CD pipelines, enabling automatic and consistent observability across environments. This ensures that issues are detected and addressed before they reach production, significantly reducing downtime and errors. For instance, commercetools, a leader in headless commerce, [adopted MaC](https://www.checklyhq.com/case-study/commercetools/) to implement continuous monitoring throughout their SDLC. By codifying their monitoring setup, they ensured every stage of their pipeline was equipped with accurate, automated checks. This approach caught issues early in development, reduced production incidents, and provided end-to-end visibility into their systems without adding extra operational overhead. ### **Automating Monitoring for Rapidly Changing Systems** Monitoring APIs and UIs in fast-paced environments is a complex challenge, especially when new endpoints are constantly being added or updated. Traditional approaches often involve manually setting up and maintaining checks, which is time-consuming and prone to errors. As systems scale, gaps in monitoring coverage or misconfigured alerts can lead to missed issues and delayed responses, impacting the reliability of your services. Monitoring as Code solves this by automating the creation and deployment of monitoring configurations, ensuring every API is consistently and comprehensively monitored from staging to production. By integrating monitoring into your development pipelines, you can keep up with rapid changes without sacrificing reliability. For example, TrueLayer, a leading open banking platform, [adopted Monitoring as Code to address these challenges](https://www.checklyhq.com/case-study/truelayer/). By codifying their API monitoring configurations and automating their setup with Checkly, they reduced manual effort by 60%. This allowed their team to scale monitoring as they introduced new endpoints, ensuring complete coverage and faster issue detection, all while focusing more on innovation rather than operational tasks. ### **Enhancing Monitoring Ownership Across Teams** In large organizations, monitoring is often centralized under SRE teams, creating bottlenecks and slowing down response times. This approach limits the ability of engineering teams to adapt and manage monitoring for their own services. Monitoring as Code (MaC) decentralizes this process, empowering teams to define and control their monitoring setups directly within their workflows. This fosters greater accountability and enables faster detection and resolution of issues. [LinkedIn leveraged MaC](https://www.checklyhq.com/case-study/modernizing-linkedins-monitoring-infrastructure/) to transition from an SRE-managed model to one where engineering teams took full ownership of their monitoring. Using Checkly, they streamlined the process, allowing teams to tailor monitoring configurations to their specific needs. This shift reduced time to detection, improved system reliability, and gave teams the autonomy to innovate without relying on a central authority. ## What Can You Monitor Using Monitoring as Code? Monitoring as Code (MaC) enables you to test key aspects of your application and infrastructure with precision and automation. Here are some of the areas where it excels: ### Application Uptime Application uptime reflects how reliably your application is available to users, and even a few minutes of downtime can lead to user frustration, loss of revenue, and damage to your reputation. MaC allows you to define uptime checks programmatically, ensuring these checks are consistent across all environments. For example, you can configure MaC to ping specific endpoints or services at regular intervals to verify they are reachable and responding within acceptable timeframes. If a failure is detected, MaC immediately triggers alerts, allowing teams to investigate and resolve issues before they escalate. MaC also simplifies scaling uptime monitoring across multiple regions. By deploying checks from different geographic locations, you can ensure your application is accessible worldwide and identify regional connectivity issues. Additionally, codified monitoring enables you to keep pace with application changes—new services and endpoints are automatically incorporated into your monitoring setup, reducing the risk of missed coverage. ### **UI Functionality** MaC allows you to simulate and test user interactions with your web application. For example, you can create browser checks to validate critical workflows like logging in, completing a checkout, or performing a search. These automated tests ensure that every step—loading pages, submitting forms, and rendering elements—works as expected. They catch issues early, safeguarding functionality and user satisfaction. With MaC, browser tests are version-controlled and evolve with your application. You can easily scale monitoring across regions, browsers, and devices, ensuring a seamless experience for all users. Here’s an example of a browser monitor that logs into Checkly, and waits for the dashboard to fully load. ```jsx theme={null} import { expect, test } from '@playwright/test' // 1 test('Login to Checkly', async ({ page }) => { // 2 await page.goto('https://app.checklyhq.com/login') // 3 await page.locator('input[type="email"]').type('john@example.com') // 4 await page.locator('input[type="password"]').type('mypassword') // 4 await page.getByRole('button', { name: 'Log In' }).click() // 5 const homeDashboardTable = page.getByTestId('home-dashboard-table') await expect(homeDashboardTable).toBeVisible() // 6 }) ``` ### **API Health** APIs power modern applications, and MaC ensures their availability, performance, and correctness. You can automate checks to validate response status codes, measure latency, and verify payloads. These checks provide confidence that your APIs function reliably under different conditions. By deploying API checks globally, you can ensure consistent performance for users across all regions. Since monitoring setups are code-based, they are easy to update as your APIs evolve. This keeps monitoring aligned with API changes, eliminating gaps in coverage. Here’s an example of an API check that tests an e-commerce flow of updating an inventory item: ```jsx theme={null} import { ApiCheck, Frequency, AssertionBuilder, RetryStrategyBuilder } from 'checkly/constructs' new ApiCheck('update-inventory-item', { name: 'update inventory item', activated: true, muted: false, shouldFail: false, runParallel: false, locations: ['eu-central-1', 'us-west-1'], tags: [], frequency: Frequency.EVERY_10M, environmentVariables: [], // group: your check belongs to group 'Inventory API', maxResponseTime: 10000, degradedResponseTime: 5000, request: { url: 'https://danube-webshop.herokuapp.com/api/books', method: 'GET', followRedirects: true, skipSSL: false, assertions: [ AssertionBuilder.jsonBody('$.length').equals(30), AssertionBuilder.statusCode().equals(200), ], body: ``, bodyType: 'NONE', headers: [], queryParameters: [], }, retryStrategy: RetryStrategyBuilder.fixedStrategy({ baseBackoffSeconds: 0, maxRetries: 1, maxDurationSeconds: 600, sameRegion: false, }), }) ``` ### Application Performance Slow app load times or high latency can drive users away and impact revenue. Monitoring as Code provides a streamlined way to track performance metrics like page load times, API response times, and latency, ensuring your application consistently meets expectations. With MaC, performance checks can be integrated into your CI/CD pipelines to catch slowdowns early in development or staging. Automated alerts notify your team of performance issues in real time, enabling proactive fixes before users are impacted. MaC also excels at monitoring deployments by running performance checks automatically after each release. This ensures that new changes don’t introduce regressions, and if issues arise, teams can act quickly to resolve or roll back, maintaining stability and trust. MaC makes it easy to monitor performance across regions, simulating user experiences in different locations. This helps identify latency caused by network issues, server configurations, or CDNs, ensuring fast, consistent performance worldwide. By codifying performance monitoring, MaC standardizes and automates the process, allowing your team to maintain a responsive, reliable web app at scale. ### **Scheduled Jobs or Cron Jobs** Scheduled jobs or cron jobs are crucial for automating tasks, but failures can lead to major problems if undetected. MaC enables [heartbeat monitoring](https://www.checklyhq.com/blog/heartbeat-monitoring-with-checkly/) to ensure these jobs run on schedule. A heartbeat monitor verifies that a job reports back within the expected timeframe. If it doesn’t, an alert is triggered, giving you a chance to fix the issue before it impacts your system. This proactive approach ensures your critical automation workflows are reliable and resilient. ### **Infrastructure Reliability** MaC also tests the health and reliability of your infrastructure. This includes uptime monitoring for servers, latency checks for databases, and validating metrics like CPU or memory usage. By codifying these checks, you create an automated system that detects potential issues early. This helps you resolve problems before they escalate into critical incidents. ## Monitoring as Code with Checkly Monitoring as Code (MaC) with Checkly empowers you to streamline, automate, and scale your monitoring strategy by leveraging advanced tools and integrations. With native Playwright support, Terraform and Pulumi integrations, and seamless CI/CD pipeline compatibility, Checkly simplifies complex monitoring setups and enables dynamic, code-driven observability. ### **CI/CD Pipeline Integration** Checkly’s compatibility with CI/CD pipelines ensures monitoring is embedded into your development workflow. You can configure performance, uptime, and functional checks to run automatically during deployments, catching issues early and maintaining system reliability. This tight integration helps teams: * Validate changes before they impact production. * Monitor deployments in real-time to detect regressions or failures. * Streamline troubleshooting by aligning monitoring data with code changes. ### Native Playwright Support With native Playwright support, Checkly enables you to directly use existing Playwright scripts as browser checks. This integration allows you to monitor complex user workflows—such as logins, form submissions, or multi-step transactions—without needing to rewrite tests. Your monitoring aligns with your application logic, offering precise, actionable insights into the user experience. ### **Alert Management via Code** A strong alerting strategy ensures critical issues are addressed promptly while minimizing noise for your team. With Checkly, you can configure alerting integrations like [PagerDuty](/docs/integrations/incident-management/pagerduty) or [OpsGenie](/docs/integrations/incident-management/opsgenie) directly in your codebase. This approach allows you to: * Define alert thresholds, escalation paths, and severity levels as part of your monitoring setup. * Automatically deploy and update alerting configurations alongside your application changes. * Maintain consistency across environments, ensuring no alerts are missed or misconfigured during rollouts. ### **Status pages and dashboards** [Status pages](/docs/communicate/status-pages/overview) and [internal dashboards](/docs/communicate/dashboards/overview) offer a real-time view of your application's health, and MaC enables you to configure these visualizations programmatically. Tools like [Grafana](/docs/platform/reporting/analytics-api#building-a-grafana-dashboard) integrate easily with MaC, allowing you to: * Automate the creation and updates of dashboards and status pages based on predefined templates. * Highlight key metrics like latency, error rates, or resource utilization, making it easier to identify trends and anomalies. * Share consistent, reliable dashboards across teams or environments. * Communicate app and API availability to your customers. ### **Performance Benchmarking** Performance benchmarking helps you understand your application’s limits and track improvements over time. MaC allows you to automate performance tests and benchmark configurations, ensuring consistent measurement across deployments. You can: * Define baseline performance metrics in your codebase. * Run automated checks during staging or production deployments to compare against benchmarks. * Continuously refine performance goals based on historical data. Codified benchmarks keep your team aligned and make it easier to identify potential bottlenecks before they impact users. ### **Incident Postmortems and Continuous Improvement** Postmortems are crucial for learning from incidents, and MaC ensures these lessons are integrated directly into your monitoring setup. After identifying root causes, you can: * Update monitoring checks and alert thresholds via code to prevent similar issues. * Document findings in version-controlled repositories, keeping configurations transparent and actionable. * Automate the deployment of improvements across all environments. This approach turns every incident into an opportunity for growth, building resilience into your application with each iteration. ### **Regional or Multi-Environment Testing** Modern applications often run across multiple regions or environments, and ensuring consistent performance everywhere is a major challenge. MaC allows you to: * Configure checks that simulate user interactions or API calls from different regions to identify latency or availability issues. * Automate regional monitoring setups, ensuring every environment, from staging to production, is monitored uniformly. * Adjust alert thresholds and benchmarks for specific regions or conditions, such as high traffic during local peak hours. This granular, automated approach ensures your application delivers a seamless experience to users regardless of their location or environment. ### **Infrastructure as Code (IaC) Support** Checkly integrates seamlessly with popular IaC tools like Terraform and Pulumi. This means you can define monitoring configurations alongside your infrastructure code, ensuring monitoring setups are automatically deployed with your infrastructure updates. These integrations enable: * Consistent monitoring deployment across all environments. * Simplified updates to monitors as your infrastructure evolves. * Centralized version control, ensuring your monitoring is always aligned with your application and infrastructure. ## Best Practices for Monitoring as Code Implementing monitoring as code effectively requires following best practices to ensure reliability, scalability, and maintainability. 1. **Modular Configurations**: Break down monitoring configurations into reusable modules. This makes it easier to manage and update configurations. 2. **Environment-Specific Settings:** Use environment-specific settings to tailor monitoring configurations based on the environment. This ensures that monitoring is relevant and accurate. 3. **Automation and Testing:** Automate the deployment of monitoring configurations and regularly test them to catch issues early. 4. **Documentation:** Maintain comprehensive documentation for all monitoring configurations and processes. This aids in onboarding new team members and maintaining consistency. 5. **Continuous Improvement:** Regularly review and update monitoring configurations to adapt to changes in the application and infrastructure. ### **Why Checkly for Monitoring as Code?** By combining native Playwright support, IaC integrations like Terraform and Pulumi, and seamless CI/CD pipeline compatibility, Checkly offers a robust, flexible, and scalable solution for Monitoring as Code. Whether it’s managing alerts, benchmarking performance, or monitoring deployments, Checkly ensures your monitoring evolves dynamically with your application, delivering reliability and efficiency at every step. [Start streamlining your monitoring strategy today](https://app.checklyhq.com/signup) with Checkly and experience the power of Monitoring as Code firsthand. # Monitoring Resources - Tools, Metrics & Best Practices Source: https://www.checklyhq.com/docs/learn/monitoring/overview Browse essential monitoring resources from Checkly. Learn key metrics, performance tools, and best practices to optimize your systems and user experience. Monitoring your production services is a key part of a good user experience. Traditional stopgaps like 'pingers' or simply waiting for user reported issues are insufficient. These guides are an introduction to the basic concepts of monitoring. ## Getting Started Bring transparency, scalability, and efficiency to your cloud infrastructure management. How does software get made? Breaking down the basic phases of software development. ## Monitor your critical systems Learn all about web application monitoring, performance and reliability and discover top tools. Discover the benefits and top tools for frontend monitoring. Track performance and optimize user experience. ## On the Checkly Blog Learn seven best practices to fight alert fatigue. Traditional monitoring has become insufficient for managing complex systems. Five best practices to help you deploy your software more securely and reliably. # Real User Monitoring (RUM) - Benefits, Challenges, and Alternatives Source: https://www.checklyhq.com/docs/learn/monitoring/real-user-monitoring Discover the benefits, challenges, and top tools for frontend monitoring. Learn how to track performance, detect issues, and optimize user experience. ## **Why Real User Monitoring Matters** Real User Monitoring (RUM) gives you actual performance data from real people using your website or app. Unlike synthetic tests (which simulate users), RUM shows you exactly what your customers experience—the promise is that if even one of your users encounters an error, you’ll know about it. ### **Key Benefits:** * **Pinpoint Performance Issues** – Discover which pages load slowly or behave poorly on specific devices (e.g., mobile vs. desktop). * **Reduce User Frustration** – Catch errors before they drive users away. * **Optimize Business Impact** – Faster pages = happier users = more conversions (sales, signups, engagement). * **Identify Trends —** Will users trigger this edge case error often? Don’t just speculate, observe! * **Prioritize Fixes** – Instead of guessing, use real data to decide what to improve first. **Example:** If checkout pages take 8 seconds to load on mobile, but only 2 seconds on desktop, RUM helps you find and fix the bottleneck. RUM monitoring shows exactly which users are encountering slow load times, and you can look at historical trends to see if a recent change may have been the cause. ## **How RUM Works: From Data to Insights** ### **1. Data Collection (The "What")** A lightweight JavaScript snippet (or mobile SDK) tracks: * **Page load times** How long until the page is usable? * **Resource timing** Did slow images or APIs delay rendering? * **User interactions** Clicks, scrolls, form submissions * **Errors** JavaScript crashes, failed API calls, broken assets All of this data is collected, generally as a trace with annotations for events and errors, for tramsission by the lightweight RUM code. ### **2. Data Transmission (The "How")** * Data is sent to a monitoring backend * **Sampling** (optional) reduces cost by sending only a % of sessions. ### **3. Analysis & Action (The "Why")** * **Dashboards** show performance trends (e.g., "Mobile users in Europe are 40% slower"). * **Alerts** notify you when error rates spike. * **Session replay** (in some tools) lets you watch real users struggle with bugs. **Pro Tip:** Start with core metrics (load time, error rate), then expand to track business goals (e.g., "How long until users can play a video?"). *** ## **Sampling & Storage: Balancing Cost and Detail** ### **The Data Fire Hose Problem** RUM generates **massive** amounts of data. If you track every click, scroll, and page load for millions of users, costs can spiral. Storing RUM data isn’t just expensive for your storage solutions, it also means incredibly slow queries for trying to see historical performance. ### **Solution: Tiered Storage (Like a "Data Lake")** 1. **Raw Bucket** – Store *everything* (for rare deep dives). * Example: Keep 100% of data for 7 days, then archive. 2. **Refined Bucket** – Keep only *key metrics* (for daily dashboards). * Example: Sample 10% of sessions to track trends. **Why This Works:** * **Engineers** can query raw data when debugging. * **Product teams** use refined data for high-level trends. **Inspiration:** This mirrors the "lakehouse" approach in big data—store first, filter later. ## The limitations of RUM The data fire hose problem mentioned above points to a more general issue with Real User Monitoring as a solution to observability problems: when compared with a measurement and testing tool using synthetics monitoring, the advantage of RUM is supposed to be that it measures every user everywhere. However as we see above the first solution to issues of storage and bandwidth is to start sampling data. Suddenly our monitoring of all users became ‘some users, sometimes.’ ### RUM and the panopticon: you can’t really see everything The issues with RUM are rather well correlated to the story of another ‘total observability’ solution, the classic panopticon. Originally conceived by Jeremy Bentham in the 1780’s, the idea is familiar to anyone who’s noticed hundreds of camera installations at their local Wal-Mart: a prison or factory where everyone could be observed at any time, but cannot see their observer. A diagram of the Real User Monitoring model

*Jeremy Bentham’s original Panopticon concept. Image by Blue [Ākāśha](https://commons.wikimedia.org/w/index.php?curid=95988220).* The promise of the panopticon was a kind of social utopia: perfect behavior since everything can be observed. While this sounds a bit silly in the 21st century, I must make the comparison to the promises of RUM: once we can observe errors experienced by every user, we can fix all the major user pain points and everything will get better. In both cases the reality faced problems of practicality: A guard in a panopticon prison can’t actually see everything in the cells he’s guarding, leading to bad behavior only in blind spots. This is comparable to the existence of user categories that don’t run RUM instrumentation well, or don’t successfully transmit data to your monitoring backend, or who experience total crashes of their browser or app: often the users having the worst experience are the ones who don’t show up in real user monitoring. A diagram of the Real User Monitoring model

*The Panopticon, which for nearly a century promised improved conditions for large institutions, proved impractical in reality. Constructions like this one in Cuba are now abandoned. Image by [Friman](https://commons.wikimedia.org/w/index.php?curid=2410607), CC BY-SA 3.0* Those monitoring providers who sell RUM services will be quick to assure that RUM can be done efficiently, often with no sampling. APM providers with large RUM offerings will promise that you can observe every user, everywhere, and send data about them as much as you like. All I’ll say to this is the people insisting RUM’s data overhead is no problem, are the same people who make money based on how much RUM data you send. *** **Ready to start low-overhead frontend monitoring today?** check out our practical guide on [how to start frontend monitoring with Checkly and Playwright.](https://www.checklyhq.com/guides/monitoring-ecommerce-apps-using-playwright/) *** ### Sampling and its discontents Since it is impractical to store every click, scroll, action, and web response for each user, real RUM implementations generally involve some form of sampling. This has some notable drawbacks: * If you’re sampling only after data storage, the data lake model mentioned above, it means the vast majority of data that you paid to transmit, and process, is getting thrown away relatively soon. Further if your RUM monitoring slows down the user experience at all, it does so for everyone. This issue is best solved by sampling at the time of data measurement, called ‘head-based sampling,’ where users are assigned to be monitored randomly as their session starts. That means we don’t use resources on a session we’re not going to save. A diagram of the Real User Monitoring model

In the diagram above, every sixth session is measured and transmitted, but as you can see in this not-uncommon example, all the slowest sessions have been discarded. * With random sampling, the chance that a specific user report won’t have RUM data available starts to increase Problem sessions are almost always edge cases, and so random sampling means that most of those unusual cases are being thrown out. What would be better is to decide at the end of a session if the data was interesting. This is the much vaunted tail-based sampling. A diagram of the Real User Monitoring model

In this example the longest user session is sent (because it was probably frustrating to the user) and the fastest (because it may contain an error), and the others discarded. In the real world things like the number of spans in the trace would also help determine which to send. Both head- and tail-based sampling are concepts that are applicable beyond real user monitoring, and in all cases tail-based sampling proves quite difficult to implement in practice. The situation with RUM is illustrative of the general problem with tail-based sampling: when a user finishes with a session, the RUM code in their browser or application only has the local context to decide if this trace should be sent. This means either arbitrary and simplistic thresholds like ‘only send sessions over X seconds’ or the implementation of yet another complex system to look at recent trends in RUM traces, decide what looks interesting, and individually inform RUM agents if they should send particular traces. An inelegant solution that **still** misses some cases and doesn’t actually monitor every user in every session! ### Another kind of sampling problem: the level of RUM measurement A key recommendation for monitoring with RUM is to start small: observe overall page load times and errors in the browser, send that data, and build from there. Eventually though we’d like to monitor lots of things about the user session: what’s being loaded, requested, what’s happening in the background, what third party plugins are doing, etc. But by implementing all this added instrumentation it’s unavoidable that we end up slowing down the user experience the more we want to observe. Since some of the most concerning user sessions are those that end either in an app crash or the user exiting the page or service, it’s further tempting to have RUM data report frequently during the user session. This may be a necessary step to ensure the most troublesome traces are sent, at least partially, but it further adds to the network overhead. In the end both in trace detail and number of sessions traced, **RUM monitoring by its nature is limited in how closely it can observe all users**. ## Conclusions: RUM is about aggregate performance, not high resolution Given the concerns about RUM in practice, it rarely succeeds in delivering the promise of detailed data on every user session. If a sales rep were to call and say: ”Jeff Roberts was logging in to view his photos yesterday and all his photos showed error messages, what happened?” It’s unlikely that Real User Monitoring, as it works in the real world, would be able to provide an answer. Does that mean that we shouldn’t bother with real user monitoring? No, RUM is a core part of monitoring your service, but it’s more about trends and collective observation than high resolution tracing. Consider questions like: * What percentage of our users experience this error? * When an image takes more than 8 seconds to load, how many users leave the page? * What’s the request that’s taking up the most *total* user time (number of requests x time taken)? These are questions that only RUM can answer easily. By focusing on aggregate insights, RUM helps prioritize optimizations that benefit the majority of users, while [synthetic monitoring](/docs/learn/monitoring/transaction-monitoring) performs consistent, repeatable observation of every critical path, and logs fill in the gaps for edge cases. The key is balancing depth with practicality—using RUM to guide improvements, not chase an unattainable "perfect observability" ideal. Start small, sample wisely, and correlate RUM data with other telemetry to build a complete picture of your application’s health. # Synthetic Monitoring - Concepts, Benefits & Challenges Source: https://www.checklyhq.com/docs/learn/monitoring/synthetic-monitoring Explore the what and why of synthetic monitoring. Synthetic monitoring is the process of proactively simulating an interaction with a website, application, or API to measure that it’s available and performant in a given scenario. Synthetic monitoring also deals with alerting the right team with as much information as possible if an error or degradation occurs. Synthetic monitoring enables software teams to better understand the performance of their application by simulating realistic scenarios, like uptime of a homepage from a location in Europe, or the performance metrics of the entire checkout process in an eCommerce application at high-frequency intervals against production. It reduces risk in your software by lowering the overall time to detect an error that could negatively impact a customer by proactively alerting teams if performance degrades. Just as automated testing is essential for high-velocity development teams, enabling them to catch bugs and defects by simulating real user flows in testing and preview environments, synthetic monitoring enables teams to continuously monitor for optimal user experiences. Some synthetic monitoring tools, like Checkly, allow engineers to reuse entire end-to-end tests as production monitors. This enables true continuous quality to be practiced by organizations. ## How does Synthetic Monitoring work? At its core, synthetic monitoring is about simulating user interactions with an application to preemptively identify issues before they impact real users. Synthetic monitoring gives developers and operations engineers the confidence that they are one step ahead, ensuring that potential problems are identified and resolved swiftly. In practice, synthetic monitoring works a lot like automated testing in production. Teams constantly run “tests” against a production environment from global locations. Using a Monitoring as Code workflow, its easy to even reuse real end-to-end tests as monitors, simply by deploying them to infrastructure such as Checkly. Synthetic monitors can measure uptime, transactions, status response performance and more - all from different regions of the globe set at a pre-configured frequency. This enables continuous confidence when making rapid changes to mission-critical services and applications by allowing you to notice issues before your customers do. ## Benefits of Synthetic Monitoring Synthetic monitoring plays a pivotal role in understanding an application's behavior under various scenarios. It allows teams to test how their application performs in different geographical locations, under various network conditions, and during peak load times. This insight is critical for applications that cater to a global audience, where performance can vary significantly based on a user's location and the time of day. Here are some other benefits of synthetic monitoring: ### Proactive error detection Considering your application's essential features, *how long would it be acceptable for these to be broken?* The answer to this question is your ideal monitoring interval. You can only fix production issues you know about. If making a purchase is your core business, you probably don't want to test this functionality only once a day after a production deployment. Nor do you want to rely on an observability platform reactive alerting you after a customer has encountered the broken checkout flow. ### Faster mean time to detection Synthetic monitoring enables you to test your core functionality daily, every hour, or even every minute. The shorter your synthetic monitoring interval, the quicker your [mean time to detect (MTTD)](https://www.checklyhq.com/learn/incidents/reduce-mttd/) will be. A short MTTD will enable you to fix production issues before your customers reach out to your support channels! ### Accurate alerting & less noise Betting on user experience testing with synthetic monitoring leads to more meaningful alerts. A failed transaction might be an issue, but your infrastructure could handle it gracefully. Alerts based on a broken user experience tell you the entire story and must be treated critically and acted on immediately. ### Performance benchmarking Another benefit of running synthetic monitoring with headless browsers is that you can monitor performance implications while testing core functionality. Is your web app fast enough for customers in Australia? Does it provide a good Core Web Vitals experience? Do core flows like the customer login become slower over time? Many things can cause performance degradation, but synthetic monitoring will unveil a slower user experience with aggregated performance metrics. You can't fix things you don't know about! ### Controlled environments Monitors are run from controlled environments with consistent browser properties and network connections to make root-cause analysis efficient and accurate. ## Synthetic monitoring vs Real User Monitoring (RUM) Real user monitoring, also called RUM or passive monitoring, is another way to analyze and monitor your application. RUM monitoring offers insights into user interactions, performance statistics, and your users' devices and locations in a way that differs greatly from synthetic monitoring. Unlike synthetic monitoring tools, RUM monitoring tools don't simulate user interactions, but rather record real user interactions. There's nothing for you to script or define. With RUM, an embedded JavaScript snippet records and reports the behavior of actual users back to you so that you can monitor and analyze your applications' performance and health. Your users are the data source! This sounds great, but there are a few challenges with RUM. | **Feature** | **Synthetic Monitoring** | **Real User Monitoring** | | ----------------------- | ------------------------------------------------------- | ------------------------------------------------------------ | | **Monitoring Approach** | Simulates realistic scenarios and interactions | Observes and monitors real user traffic | | **Monitoring Type** | Proactively monitors for errors or degraded performance | Reactively monitors metrics for abnormal data or performance | | **Trace Generation** | Only produces traces when an error is encountered | Traces & stores vast amounts of user data (requires storage) | | **Data Privacy** | Safe, secure data used during synthetic runs | May accidentally contain private user information or PII | A comparison of RUM vs. Synthetics ## Types of Synthetic Monitoring There are a number of terms used around synthetic monitoring; everything from "Synthetic User Monitoring" to "Heartbeat Monitors" can refer to the same test! However, in general, there are four types of monitoring that are most critical for monitoring your production services: ### Uptime or Availability Monitoring Availability Monitoring is fundamental. Its primary objective is to verify whether a web service or application is accessible at any given time. This type of monitoring simulates user interactions to check the availability of websites, APIs, and servers. It's not just about confirming that a server is up and returning "200 OK" status messages; it's about ensuring that the application is operational and responding as expected. ### Transaction Monitoring Transaction Monitoring takes a step further. To know that every part is really working as expected, Transaction Monitoring means simulating complex user transactions or workflows to verify that critical processes, like checkout or login, are working as intended. Scripts are designed to mimic the path a user would take through the application. This can be as simple as logging in or as complex as completing a multi-step transaction. The goal is to identify performance issues or bugs that might not be evident in other forms of testing. For example, this will reveal whether third-party services have failed, resulting in degraded performance on our own site. ### Performance Monitoring Web Performance Monitoring is all about speed and efficiency. It evaluates how quickly and smoothly your web application loads and operates from the user's perspective. This is a more analog measurement than the Availability or Transaction Monitoring, as exact loading times will vary and won't give simple "up or down" feedback. Web Performance Monitoring can identify technical debt, as small changes to your experience can add up to degraded (or improved!) performance for your end users. ### Keyword Monitoring Keyword monitoring entails monitoring for content that is expected to appear in an API response or within a UI. [Keyword monitoring](https://www.checklyhq.com/guides/keyword-monitoring/) can be extremely helpful for commerce sites or other content-driven websites like news publishers and blogs. ## Synthetic Monitoring Best Practices Implementing synthetic monitoring effectively ensures you maximize its value in identifying and resolving performance issues. Here are key best practices to follow: ### 1. Define Key User Journeys Before you implement synthetic monitoring, identify the most critical user interactions, such as login flows, checkout processes, or API endpoints that drive your business. These are the flows you want to be setting synthetic monitors for. Simulating these core pathways ensures that issues affecting customer satisfaction are detected early. ### 2. Test From Multiple Locations Set up monitors across geographically distributed data centers to simulate user access from various regions. This practice helps uncover location-specific latency or connectivity issues and ensures a consistent experience for global users. ### 3. Use Realistic Scenarios Craft test scripts that mimic real user behaviors, including complex interactions like multi-step forms or dynamic content loading. Ensure these scripts reflect typical device types, browsers, and network conditions to emulate authentic experiences. ### 4. Monitor Continuously and Strategically Schedule monitoring at frequent intervals to capture real-time performance data. Combine this with strategic off-peak monitoring to identify time-sensitive bottlenecks, such as overnight updates or backups affecting availability. ### 5. Set Up Meaningful Alerts Configure alerts to balance urgency and relevance. Alerts should only trigger for significant issues, avoiding noise from false positives or minor deviations, to ensure your team focuses on critical problems. ### 6. Regularly Update Scripts Keep monitoring scripts updated to align with application changes. Regular maintenance prevents outdated tests from providing inaccurate results or missing new vulnerabilities. ### 7. Integrate Synthetic Monitoring with Your CI/CD Pipeline Incorporate synthetic monitoring checks into your deployment workflows to catch issues before they reach production. Automated tests can act as gatekeepers for performance and functionality. By following these best practices, you can leverage synthetic monitoring to proactively enhance performance and deliver seamless user experiences. ## Synthetic Monitoring Use Cases Here are key use cases where synthetic monitoring shines: * Website Performance Monitoring: Simulate user interactions to measure load times, responsiveness, and overall performance of web pages. * API Monitoring: Ensure APIs are functional and responsive and deliver accurate data to dependent applications or services. * Global Availability Testing: Verify the availability and performance of your applications from various geographical locations. * End-to-End Testing: Test critical user workflows, such as account creation, login, or checkout processes, to ensure seamless experiences. * Third-Party Service Validation: Monitor the performance of integrated third-party services, such as payment gateways or content delivery networks (CDNs). * Mobile App Monitoring: Test app functionalities across different devices, platforms, and network conditions. * Baseline and SLA Verification: Establish performance baselines and ensure compliance with service-level agreements (SLAs). These use cases make synthetic monitoring essential for delivering reliable and high-performing applications. ## Common Synthetic Monitoring tools To do synthetic monitoring right, you'll either need to run a service totally independently of your existing tech stack or use a SaaS tool. A system to monitor your production services shouldn't go down when your own system does! You'll also need a tool to store past performance data and present it within a dashboard. ### Playwright - The leader in browser automation Backed by Microsoft, Playwright quickly became one of the leaders in synthetic and [end-to-end testing](https://www.checklyhq.com/guides/end-to-end-monitoring/). Its ability to control headless browsers while providing a stellar developer experience convinced the developers, quality assurance, and DevOps communities. And we at Checkly believe it's the best tool for synthetic monitoring. ### Checkly - The best platform for running Synthetic Tests While multiple Observability vendors offer limited site pings as part of a larger toolset, Checkly offers the best experience for users focused on synthetics testing. While other teams use Playwright only for the occasional end-to-end checks, Checkly lets you run these detailed checks with complex scripted behavior at any cadence you need to maintain your SLA. And Checkly can cost less than [half of what our competitors charge](https://www.checklyhq.com/blog/how-to-spend-ten-grand-12-bucks-at-a-time/). Checkly offers rich options for scheduling tests on a cadence, including [scheduling checks from multiple regions](https://www.checklyhq.com/blog/parallel-scheduling/), and offers the most complete answer to ‘how is our service performing for all our users? ### Monitoring as Code At Checkly, we’ve pioneered Monitoring as Code to help shift observability left and unite testing and monitoring in the SDLC. With MaC, you can reuse real, end-to-end Playwright tests as monitors, configure alerting and thresholds with code, and deploy Dashboards and Status Pages to communicate your application’s health with internal and external stakeholders. ## Conclusion Synthetic monitoring forms the foundation of shipping a seamless user experience. It is a safety net for development and DevOps teams, allowing them to innovate confidently. By integrating synthetic monitoring with a MaC approach, we create a bridge between testing and monitoring, fostering a collaborative environment that enhances the overall health of your application. Synthetic monitoring and Monitoring as Code anticipate and resolve issues before they impact users and streamlines the process of maintaining a high-performing, user-centric application. In the fast-paced world of development, synthetic monitoring and MaC are tools and essential practices that ensure your application consistently delivers an optimal user experience. Because you have to remember you can only fix issues you know about. ## Frequently Asked questions about Synthetic Monitoring ### What is the difference between observability and synthetic monitoring? The concepts of observability and monitoring aim to provide critical insights into an application's health and performance. And while both seem similar, they differ in their approach to system analysis. Synthetic monitoring, on the one hand, operates on a set of predefined actions leading to certain results and metrics. It includes calling API endpoints to check for correct and performant responses or controlling headless browsers to simulate user actions and test for functioning UI interactions. Synthetic monitoring looks at your application from the outside and constantly tests that everything operates correctly. Observability, on the other hand, helps to analyze a system’s internals by scanning and parsing log files, traces, and system events. Thanks to observability you are able to trace events across system boundaries and services. It enables a comprehensive view of your infrastructure, the taken operations, and your system’s overall health and performance. Both approaches are valuable to guarantee a well-functioning and highly available system, but only in combination do they really shine. Synthetic monitoring enables you to proactively test your application's functionality and get alerted when something is wrong and impacts end user experience. Observability then enables you to find out what’s wrong after getting alerted. ### How does synthetic monitoring contribute to maintaining website uptime? To ensure that your website and APIs have high availability, synthetic monitoring enables you to constantly test your website’s uptime and the provided functionality. By scheduling automated API calls and headless browser sessions, you can be assured that your website is up and running at any time and from anywhere in the world. Synthetic monitoring is a proactive approach to ensure your application works as expected and enables you to receive timely alerts in case something is wrong. ### What advantages does synthetic monitoring have over older monitoring techniques? Contrary to older and passive monitoring methods, synthetic monitoring enables you to be proactive and leverage automation scripts to test your application and infrastructure. Make API calls or simulated browser sessions to constantly test and monitor your application, gather performance metrics, and most importantly be certain that your application and the underlying infrastructure perform like they’re supposed to. Thanks to synthetic monitoring you’ll get alerted and know about issues before your customers do! ### How do passive and synthetic monitoring differ? Passive monitoring observes and monitors existing traffic and user interactions. It’s a reactive approach to analyze performed actions and traffic to identify potential issues. Passive monitoring looks at and analyzes existing traffic and it doesn’t lead to any additional requests or browser sessions. Synthetic or active monitoring simulates user actions such as making API calls or performing a browser session with a real browser. Synthetic monitoring is a proactive approach to test if an application is working correctly globally at all times. ### How are performance metrics specifically captured and analyzed in synthetic monitoring? Synthetic monitoring enables the collection and capturing of critical performance metrics. Synthetic [API monitoring](https://www.checklyhq.com/blog/what-is-api-monitoring/) performs your predefined API calls and collects performance metrics such as Time to first byte (TTFB) or DNS resolution time. It enables you to define thresholds to capture performance regressions and get alerted when your APIs become slower over time. Synthetic end-to-end monitoring simulates real browser sessions that perform application user flows like buying a product or logging into an account. While navigating your website, common frontend performance metrics such as load time or core web vitals such as largest contentful paint (LCP) and total blocking time (TBT) will be gathered and visualized to discover slow-performing frontends. Visualized and captured performance metrics on the API and UI level provide clear insights into user experience and allow you to notice and prevent performance regressions before they become an issue. # TCP and SSL - What an application developer needs to know Source: https://www.checklyhq.com/docs/learn/monitoring/tcp-and-ssl As an application developer, you need to know how TCP and SSL/TLS work together to establish secure connections, authenticate parties, and ensure data integrity. This section will explain the relationship between TCP and SSL/TLS, focusing on authentication and how it impacts your application. ## Introduction When building secure applications, understanding how TCP works with SSL/TLS (Secure Sockets Layer/Transport Layer Security) is crucial. SSL/TLS is a cryptographic protocol that provides secure communication over a TCP connection. It ensures that data transmitted between a client and server is encrypted and authenticated, protecting it from eavesdropping, tampering, and forgery. *** One senior developer shared a story of why it's important to understand the networking aspects of production service maintenance: "Early in my career, I had to learn about load balancers the hard way. A senior engineer left, and as the junior, I was left to figure things out. At the time, I didn’t fully understand SSL connections. I made a change to a VIP (Virtual IP) that handled all our Point of Sale transactions. I switched it from a non-SSL setup to an SSL one—but I forgot to add the proper certificates. Suddenly, our entire retail system went down. For an hour, I had no idea I was the cause. Then, while checking reports, I noticed tons of dropped connections. I realized my mistake, reverted the change, and everything went back to normal. The impact? About \$11 million in revenue was lost because transactions couldn’t be processed. Ouch . This taught me why understanding SSL connections is critical. A small mistake can have huge consequences. Always double-check certificates and configurations—especially when dealing with critical systems." *** As an application developer, you need to know how TCP and SSL/TLS work together to establish secure connections, authenticate parties, and ensure data integrity. This article will explain the relationship between TCP and SSL/TLS, focusing on authentication and how it impacts your application. ## How TCP and SSL/TLS Work Together TCP provides the underlying communication channel for SSL/TLS. Here’s how they work together: TCP Connection Establishment: First, a TCP connection is established using the three-way handshake (SYN, SYN-ACK, ACK). This creates a reliable communication channel between the client and server. SSL/TLS Handshake: Once the TCP connection is established, the SSL/TLS handshake begins. This process negotiates encryption algorithms, authenticates the server (and optionally the client), and establishes shared encryption keys. Secure Data Transmission: After the SSL/TLS handshake, all data transmitted over the TCP connection is encrypted and secure. The SSL/TLS handshake is critical for authentication, ensuring that the client is communicating with the intended server and not an imposter. ## SSL/TLS Authentication Authentication in SSL/TLS ensures that the parties involved in the communication are who they claim to be. This is typically done using digital certificates and public-key cryptography. ### Server Authentication In most cases, SSL/TLS is used to authenticate the server to the client. Here’s how it works: Server Certificate: The server presents its digital certificate to the client during the SSL/TLS handshake. This certificate contains the server’s public key and is signed by a trusted Certificate Authority (CA). Certificate Validation: The client verifies the server’s certificate by checking: * The certificate’s validity period (not expired). * The certificate’s signature (signed by a trusted CA). * The server’s hostname matches the certificate’s Common Name (CN) or Subject Alternative Name (SAN). Key Exchange: If the certificate is valid, the client uses the server’s public key to encrypt a shared secret (e.g., a pre-master secret) that is used to generate encryption keys for the session. This process ensures that the client is communicating with the legitimate server and not a man-in-the-middle attacker. ### Client Authentication (Optional) In some cases, the server may also require the client to authenticate itself. This is less common but is used in scenarios where mutual trust is required (e.g., enterprise applications, APIs). Here’s how it works: Client Certificate: The client presents its digital certificate to the server during the SSL/TLS handshake. Certificate Validation: The server verifies the client’s certificate, checking its validity, signature, and other attributes. Mutual Trust: If the client’s certificate is valid, the server proceeds with the handshake. Otherwise, the connection is terminated. Client authentication adds an extra layer of security but requires managing client certificates, which can be complex. *** ## SSL/TLS Handshake in Detail The SSL/TLS handshake is a multi-step process that establishes a secure connection. Here’s a simplified overview:

After a TCP handshake is complete, at the ACK step: 1. Client Hello: The client sends a "Client Hello" message to the server, specifying supported SSL/TLS versions, cipher suites, and a random byte string. 2. Server Hello: The server responds with a "Server Hello" message, selecting the SSL/TLS version, cipher suite, and providing its own random byte string. 3. Server Certificate: The server sends its digital certificate to the client. Key Exchange: The server may send additional key exchange information (e.g., Diffie-Hellman parameters) depending on the cipher suite. 4. Client Key Exchange: The client generates a pre-master secret, encrypts it with the server’s public key, and sends it to the server. 5. Certificate Verify (Optional): If client authentication is required, the client sends its certificate and a signature to prove ownership of the private key. 6. Finished Messages: Both the client and server exchange "Finished" messages to verify that the handshake was successful and that the keys are correct. Once the handshake is complete, the secure connection is established, and encrypted data transmission begins. *** ## What Application Developers Need to Know Certificate Management * Server Certificates: Ensure your server has a valid certificate signed by a trusted CA. Self-signed certificates are not trusted by default and may cause errors in clients. * Client Certificates: If your application requires client authentication, you’ll need to manage client certificates, including issuing, distributing, and revoking them. SSL/TLS Configuration * Use strong cipher suites and disable weak ones (e.g., SSLv2, SSLv3, and weak ciphers like RC4). * Keep your SSL/TLS libraries up to date to protect against vulnerabilities (e.g., Heartbleed, POODLE). Hostname Verification * Always verify that the server’s hostname matches the certificate’s CN or SAN. Skipping this step can expose your application to man-in-the-middle attacks. Error Handling * Handle SSL/TLS errors gracefully. Common errors include expired certificates, mismatched hostnames, and untrusted CAs. Provide clear error messages to users or logs. Performance Considerations * The SSL/TLS handshake adds latency to the connection establishment process. Use techniques like session resumption or TLS False Start to reduce handshake overhead. * Offload SSL/TLS processing to dedicated hardware or services (e.g., load balancers) if your application handles high traffic. *** ## Common Issues and Troubleshooting Certificate Expiry * Certificates have a limited validity period. If a certificate expires, the SSL/TLS handshake will fail. Monitor certificate expiration dates and renew them before they expire. Untrusted Certificates * If the client does not trust the CA that signed the server’s certificate, the connection will fail. Ensure your server’s certificate is signed by a widely trusted CA. Mismatched Hostnames * If the server’s hostname does not match the certificate’s CN or SAN, the client will reject the connection. Double-check your certificate’s hostname configuration. Weak Cipher Suites * Using weak cipher suites can expose your application to attacks. Regularly review and update your SSL/TLS configuration to disable weak ciphers. Handshake Failures * Handshake failures can occur due to mismatched SSL/TLS versions, unsupported cipher suites, or network issues. Use tools like `openssl` or `ssllabs.com` to diagnose handshake problems. *** ## Best Practices for SSL/TLS Authentication * Use Strong Certificates: * Use certificates with a key length of at least 2048 bits (RSA) or 256 bits (ECDSA). * Prefer certificates from widely trusted CAs. * Enable HTTPS Everywhere: * Use SSL/TLS for all communication, not just sensitive data. This prevents attackers from intercepting or tampering with any part of the communication. * Implement HSTS: * Use HTTP Strict Transport Security (HSTS) to enforce HTTPS and prevent downgrade attacks. * Monitor and Rotate Certificates: * Use automated tools to monitor certificate expiration and rotate them before they expire. * Test Your Configuration: * Regularly test your SSL/TLS configuration using tools like SSL Labs’ SSL Test or `openssl` to ensure it meets security best practices. *** ## Conclusion TCP and SSL/TLS work together to provide secure, reliable communication for your application. As an application developer, understanding how SSL/TLS authentication works is essential for building secure systems. By managing certificates properly, configuring SSL/TLS securely, and handling errors gracefully, you can ensure that your application protects user data and maintains trust. SSL/TLS adds complexity to your application, but the security benefits far outweigh the effort. Follow best practices, stay informed about vulnerabilities, and regularly update your SSL/TLS configuration to keep your application secure. # Transaction Monitoring - Components, Benefits & Challenges Source: https://www.checklyhq.com/docs/learn/monitoring/transaction-monitoring Explore transaction monitoring for application developers and SREs. Synthetic transaction monitoring is a proactive approach to assessing the performance and reliability of applications by simulating user transactions. These simulations replicate real-world interactions, such as logging into an account, performing a search, or completing a checkout process, allowing organizations to detect issues before they impact actual users. Unlike traditional monitoring of a site or service, which relies on observing real user activities, synthetic transaction monitoring generates pre-defined transactions to test specific workflows, ensuring consistent and repeatable measurements. Synthetic transaction monitoring has distinct advantages when compared to Real User Monitoring for frontend applications, and requires work to integrate with other observability tools. *** **Ready to start implementing Synthetic Transaction Monitoring today?** check out our practical guide on [how to start synthetic transaction monitoring with Checkly](https://www.checklyhq.com/guides/monitoring-ecommerce-apps-using-playwright/) *** ## Synthetic Monitoring vs. Synthetic Transaction Monitoring First some language confusion: ‘Synthetic Monitoring’ isn’t a particularly widely used term, what’s with the addition of ‘transaction?’ [Synthetic monitoring](https://www.checklyhq.com/blog/what-is-synthetic-monitoring/) broadly refers to any monitoring that uses automated tests to emulate users, sometimes a simple as a page load. Synthetic transaction monitoring is a subset focused on multi-step workflows or transactions, such as filling out forms, completing e-commerce purchases, or using APIs. While synthetic monitoring may involve simple checks like testing website uptime, synthetic transaction monitoring goes deeper into the user experience by validating the performance and functionality of complex processes. ## How Does Synthetic Transaction Monitoring Work? Synthetic transaction monitoring involves several key steps: 1. **Defining Monitoring Requirements**: Product owners, support teams, and product managers define the user paths that are most crucial for the user’s experience. These should be defined as simply as possible. 2. **Script Development**: Engineers or testers write scripts defining the transactions to be monitored. These scripts simulate user actions like clicks, inputs, and navigation. Ideally a modern framework should be used to write these scripts, like [Playwright](https://www.checklyhq.com/learn/playwright/what-is-playwright/). 3. **Test Execution**: A synthetic monitoring engine executes these scripts at regular intervals, either from a single location or multiple geographies. Ideally, test execution should be run on a service well separated from your core services; you don’t want your monitoring to go down at the same time as your site! 4. **Data Collection**: Performance metrics, including response times, success rates, and error codes, are gathered during test runs. 5. **Analysis and Reporting**: Results are analyzed to identify anomalies, trends, or potential bottlenecks. 6. **Alerting**: If thresholds are breached, alerts notify relevant teams for investigation and resolution. For almost all monitoring, alerts should wait on at least one retry, so as not to cause alert fatigue. ## Why Synthetic Transaction Monitoring Matters Synthetic transaction monitoring offers several advantages, especially over real user monitoring. * **Proactive Issue Detection**: Real user monitoring fundamentally relies on a user to have a Identifying problems before real users are impacted reduces downtime and enhances user satisfaction. * **Benchmarking and Trend Analysis**: Regular monitoring helps establish performance baselines and detect deviations over time. * **Geographic Performance Insights**: Monitoring from various locations ensures applications deliver consistent experiences globally. * **Validation of Updates**: Testing new features or patches in synthetic environments ensures they function as intended before deployment. However, challenges include maintaining up-to-date scripts as applications evolve and ensuring synthetic tests accurately reflect real-world usage. ## Components of Synthetic Transaction Monitoring Effective synthetic transaction monitoring relies on several core components: **Synthetic Transaction Engine** The synthetic transaction engine executes predefined scripts, mimicking user interactions. It must handle complex workflows involving multiple steps, conditional logic, and diverse input data. At Checkly, this is a system of ‘[Checkly runners](/docs/detect/synthetic-monitoring/overview)’ that execute checks against your service at your selected cadence and geographic regions. **Scripting Language** A flexible scripting language allows testers to define transactions. Languages like JavaScript, Python, or typescript are commonly used, with support for [integrating APIs](https://www.checklyhq.com/blog/what-is-api-monitoring/), [handling cookies](https://www.checklyhq.com/learn/playwright/authentication/), and [managing dynamic content](https://www.checklyhq.com/learn/playwright/iframe-interaction/). **Test Execution Scheduler** The scheduler automates the execution of scripts at specified intervals. This ensures continuous monitoring and provides insights into performance trends over time. At Checkly this is controlled either [via the web UI](/docs/detect/testing/overview) or via our [Monitoring as Code](https://www.checklyhq.com/guides/monitoring-as-code/#) process. **Test Results Analyzer** The analyzer processes collected data, visualizing metrics like response times, error rates, and transaction success rates. It helps teams identify patterns, anomalies, and potential issues. For most use cases test results won’t require a complex analysis since pass-fail standards should be clearly defined in the scripting language. There are some edge cases like [soft assertions in Playwright](https://playwright.dev/docs/test-assertions#soft-assertions), but still a single check should be quickly to categorize as passing or failing. **Alerting System** An alerting system notifies teams when performance thresholds are breached. Alerts can be configured for various conditions, such as increased latency, failed transactions, or regional performance degradation. **Integration with Other Tools** Synthetic transaction monitoring tools often integrate with observability platforms, incident management systems, and CI/CD pipelines, enabling streamlined workflows and comprehensive insights. At Checkly, the results of synthetics checks can integrate with your OpenTelemetry data via [Checkly Traces](/docs/resolve/traces/how-it-works). ## Synthetic Transaction Monitoring Use Cases with Examples 1. **E-Commerce Checkout Process** A retailer uses synthetic transaction monitoring to simulate a user adding items to a cart, entering payment details, and completing a purchase. This ensures the checkout process remains functional, even as the underlying datastores and visual polish on the e-commrce experience change. 2. **Banking Application Login** A financial institution monitors its login functionality by simulating user logins with synthetic scripts. This helps identify authentication delays or errors before customers are affected. 3. **API Performance Monitoring**: A SaaS company tests its public APIs by simulating client requests. This ensures consistent response times and detects any unexpected behaviors. 4. **Geographic-specific Testing**: A video streaming service tests its platform from multiple regions to ensure video playback performance meets user expectations globally. 5. **Visual Regression Monitoring** A System for monitoring every pixel of a frontend experience. Generally visual regression monitoring is both more brittle and less user-centric than some of the examples mentioned above. After all, your users aren’t logging in to your site expecting certain typefaces or exact color shades. These things do matter, however, so [with Playwright you can monitor for visual regressions](/docs/detect/synthetic-monitoring/browser-checks/visual-regressions) as an advanced test type. Synthetic transaction monitoring is an essential tool for organizations aiming to ensure optimal performance and reliability of their digital services. By proactively addressing issues and gaining insights into user workflows, businesses can deliver superior experiences to their customers. ## Benefits of Synthetic Transaction Monitoring **Early Detection of Issues** Years ago I was the on-call engineer during a holiday weekend at an ecommerce service. I kept an eye on system performance and I saw good things: latency was low and recorded errors were way down. Unfortunately, it turned out that network config had locked out thousands of users. Our performance was great because most people couldn’t reach the site. While internal measurements of system performance are useful tools for measuring internal efficiency, they aren’t great at measuring your users’ experience. The end result is a system where many failure states will only be detected by your users’ reports. Synthetic transaction monitoring allows teams to identify problems before users experience them. By simulating user interactions, businesses can proactively detect performance degradation, outages, or errors in critical workflows. This minimizes downtime and helps maintain service reliability. After all, isn’t it better to be able answer every user reported issue with ‘we’re already working on it?’ **Proactive Approach to Monitoring** Rather than waiting for user-reported issues, synthetic monitoring enables a proactive approach by constantly testing systems in real-time. We can also use synthetic transaction monitoring to find trending issues: if the load times for certain actions is slowly increasing, we can identify that problem with synthetics before it causes outright errors for users. This ensures that potential issues are addressed promptly, leading to improved user satisfaction and trust in the platform. **Performance Monitoring** Synthetic monitoring tracks key performance metrics like response time, latency, and availability. By generating consistent and repeatable test scenarios, it provides actionable insights into the health of applications and services under varying conditions. Performance like response time shouldn’t be used as hard limits for passing a test, but these metrics are still generated on synthetics checks, and you can use [Checkly's 'degraded state'](https://www.checklyhq.com/docs/detect/synthetic-monitoring/multistep-checks/degraded-states) to find user paths that are responding slowly without needing to alert teams as if there were an outage. **Finding Third-party Changes** While it’s nice to imagine our service or website as self-contained, albeit with multiple internal microservices, the modern web app has more dependencies than ever, including dependencies on third party services all the way to the front end. A missing npm package or a service outage from Intercom can all cause our user experience to shift in unpredictable ways. Detecting any problems with third-party services before your users do is one of the many benefits of synthetic transaction monitoring. Some useful checks to add: * End-to-end testing of key user transaction to make sure, for example, that users can find items, add them to a cart, and check out on your e-commerce service. * [Visual regression tests](https://www.checklyhq.com/blog/visual-regression-testing-with-playwright/) to make sure third-party javascript add ons haven’t caused problems for your front end experience. * Synthetics checks pointed directly at the third party services. If responses from the your payment provider are often unreliable, set up a monitor (and an alert) to know as soon as it has a chance to affect your users. **Compliance** Synthetic transaction monitoring can aid in compliance with service-level agreements (SLAs) by consistently measuring uptime and performance against defined benchmarks. This ensures businesses meet contractual obligations and regulatory requirements. You can also use synthetic transaction monitoring to make sure you’re compliant with accessibility requirements, [in Playwright you can use the Axe testing engine to automatically check your pages for accessibility issues](https://www.checklyhq.com/blog/integrating-accessibility-checks-in-playwright-tes/). ## Challenges of Synthetic Transaction Monitoring **Need for Validation** In the early days of my career I was working on a telephone automation system; I was nearly done when I asked a sales person if our business often received inbound calls. It turned out that since our userbase was older Americans, the majority of our sales were actually made by site visitors calling us on the phone. If I had been designing synthetic monitoring for my service, my lack of knowledge of real user behavior would have made effective monitoring difficult. Although synthetic monitoring provides simulated data, it cannot replace actual user feedback. Synthetic tests assume predefined workflows and may miss issues that occur in real-world usage scenarios. Either by directly observing users interacting with your service, or detailed analytics, you need to see how users *actually* use your service. **Core Problem Analysis** When [retries are set up correctly](/docs/communicate/alerts/overview), we should only get a synthetic transaction monitoring alert when there’s a real problem that users will notice. But detecting an issue through synthetic monitoring is only the first step. Pinpointing the root cause of problems requires additional diagnostic tools and processes. Synthetic monitoring should be run as a separate service from all of your internal services and tools, and as such will lack insight into how your back end is handling requests. Once you’ve detected a problem, backend logs, traces, and other information from something like an [OpenTelemetry monitoring system](https://www.checklyhq.com/learn/opentelemetry/getting-started-with-observability/), will be needed to find a root cause. *Checkly Traces can connect synthetic site checks with the related traces observed by your [OpenTelemetry monitoring](https://www.checklyhq.com/learn/opentelemetry/).* **Tool Integration Needs** Integrating synthetic monitoring tools with existing systems can pose challenges, especially if the ecosystem includes multiple platforms and technologies. Seamless integration is crucial for accurate monitoring and efficient workflows, but it often requires customization and ongoing maintenance. As ‘table stakes’ your synthetic transaction monitoring tool should make its results available on a shared dashboqard, and route alerts intelligently either natively or through integration with tools like [Rootly](/docs/integrations/incident-management/rootly) and [Pagerduty](/docs/integrations/incident-management/pagerduty). Ideally your synthetic monitoring could automatically connect back end tracing that was started by a synthetics request. With [Checkly Traces](/docs/resolve/traces/how-it-works), you can do just that: integrate back end traces with synthetic transaction monitoring, adding insight into the root cause of errors. By understanding these benefits and challenges, organizations can better implement synthetic transaction monitoring as part of a balanced observability strategy. ## Best Practices for Synthetic Transaction Monitoring **Defining Clear Objectives** Synthetic transaction monitoring begins with a clear understanding of what your users expect. You also need to define the retry and cadence expectations, starting with your SLA with your users. **Crafting Realistic Test Scenarios** Design test scenarios that mimic real-world user interactions as closely as possible. This includes simulating different devices, browsers, and network conditions to ensure comprehensive coverage. In playwright, use the [user agent](https://www.checklyhq.com/learn/playwright/challenging-flows/#bot-detection) and [viewport](https://www.checklyhq.com/learn/playwright/taking-screenshots/#generating-and-saving-screenshots) settings to simulate different scenarios. **Prioritizing Business-Critical Transactions** Focus monitoring efforts on transactions that directly impact your business, such as login flows, checkout processes, or API integrations. Prioritizing these ensures you catch issues that could significantly affect your customers and revenue. This matters not just for the infrastructure costs of running synthetic transaction monitoring, but alert fatigue for your operations or SRE teams. **Balancing Test Frequency and Resources** Set a monitoring cadence that balances timely issue detection with efficient use of resources. High-frequency tests may be necessary for critical applications, but over-monitoring can lead to unnecessary costs. On Checkly, make sure your geographic locations make sense for your userbase. **Setting Actionable Alert Thresholds** Define alert thresholds that reflect acceptable performance levels for your business. Avoid overly sensitive thresholds that generate false positives, and ensure alerts are actionable by including relevant context for troubleshooting. Avoid hard waits and other rigid standards that result in brittle testing. **Regular Review and Update of Test Scripts and Golden Files** Test scripts should evolve alongside your application. Regularly review and update them to account for new features, UI changes, or updates to external dependencies. This helps maintain the relevance and accuracy of your monitoring. You’ll also maintain a set of ‘golden files’ that define ideal responses, either as stored responses (for example a saved JSON response from an API) or screenshots where everything is loading correctly. With Checkly, you can [update these golden files with a single command from the Checkly CLI](https://www.checklyhq.com/cli/overview). **Leveraging Integration with Observability Tools** Integrate synthetic monitoring with broader observability platforms to enhance troubleshooting and root cause analysis. Connections to logging, tracing, and metrics tools can help paint a complete picture of your application’s health. **Using Synthetic Monitoring for SLA Validation** Synthetic monitoring is a powerful tool for validating Service Level Agreements (SLAs). By continuously testing key metrics, you can provide objective evidence of SLA adherence to stakeholders. ## Synthetic Transaction Monitoring Tools Numerous tools are available to implement synthetic transaction monitoring effectively. Solutions like Catchpoint and Dynatrace offer advanced features such as global test locations, detailed reporting, and integration with other observability tools. Open-source options, such as those leveraging CNCF projects, can provide flexibility and customization for Kubernetes-based environments. **Checkly’s Role in Synthetic Monitoring** [Checkly](https://www.checklyhq.com/product/synthetic-monitoring/) is a purpose-built platform for synthetic monitoring that simplifies the creation and management of test scripts. With features like a developer-friendly CLI, seamless integration with CI/CD pipelines, and out-of-the-box support for the multi-language Playwright automation framework. Checkly enables teams to monitor critical user journeys effectively. Its dashboards and alerting systems ensure you’re always aware of issues before they impact customers. ## How Checkly Can Help with Synthetic Transaction Monitoring Checkly provides a robust set of tools to implement synthetic monitoring seamlessly. Its scripting capabilities allow you to craft realistic user scenarios while its cloud-based infrastructure ensures consistent monitoring from multiple geographies. Integration with observability platforms and APIs enables you to incorporate Checkly into your existing workflows, making it an efficient choice for synthetic monitoring needs. ## Conclusion Synthetic transaction monitoring is a critical practice for ensuring application reliability and user satisfaction. By following best practices and leveraging tools like Checkly, organizations can gain actionable insights and proactively address performance issues. Combining synthetic monitoring with real user data and broader observability efforts enhances its value, enabling teams to maintain high service quality and meet business objectives. # Web Application Monitoring - Types, Benefits & Top 10 Tools Source: https://www.checklyhq.com/docs/learn/monitoring/web-application-monitoring Explore web application monitoring to boost performance and reliability with real user insights, performance tracking, and top tools. Web application monitoring refers to the practice of observing, tracking, and managing the performance, availability, and reliability of web applications. It ensures users have a seamless experience and minimizes disruptions by identifying issues in real-time. All that’s required for something to be a ‘web application’ is that it’s more than a static site, and that it takes requests via the internet. Even an API mainly accessed via TCP is a web application! *** **Ready to start web application monitoring today?** check out our practical guide on [how to start web application monitoring with Checkly](https://www.checklyhq.com/guides/keyword-monitoring/) *** Web application monitoring is a highly generalized term, and distinctions between, for example, ‘metrics’ and ‘monitoring;’ and ‘error tracking’ versus ‘performance monitoring’ are often distinctions without a difference. The question we’re trying to answer with Web Application Monitoring is: “how well is our application performing for users?” Observations that don’t relate to this question, for example the popularity of a single post on our social media site, or how well our live-generated site follows brand standards, are outside the scope of web application monitoring. ## How Web Application Monitoring Works Monitoring tools collect data from your web application, servers, and user interactions. They process this data to generate insights about application health, performance, and user behavior. Alerts, dashboards, and reports provide actionable insights for resolving issues and optimizing performance. ## Types of Web Application Monitoring Monitoring a web application requires understanding various dimensions of performance, usability, and security. Each type of monitoring addresses specific aspects of the application to ensure seamless operation. Let’s explore these types in greater detail. **Synthetic Monitoring** [Synthetic monitoring](https://www.checklyhq.com/docs/) uses pre-recorded scripts to simulate user interactions with your web application. By performing these synthetic transactions, you can test key functionalities such as page load times, form submissions, or API calls without relying on real user activity. * Benefits: It is proactive, enabling teams to detect and resolve issues before users are impacted. Synthetic monitoring is particularly effective for testing uptime, availability during off-hours, and the impact of new deployments. * Example Use Case: Running tests on an e-commerce checkout page to ensure it processes payments correctly after a new update. * Challenges: It doesn’t capture real-world user behaviors, so it should complement, not replace, real user monitoring. See our complete guide to Synthetic Monitoring for a deeper dive. *** **Real User Monitoring (RUM)** Real User Monitoring (RUM) captures real-time data from actual users as they interact with your application. By embedding a lightweight tracking code into the application, RUM collects metrics like page load times, errors, and user interactions. * Benefits: It provides insights into actual user experiences, helping identify regional performance variations, device-specific issues, or areas of improvement in UX design. * Example Use Case: Tracking how mobile users from Europe experience an application compared to desktop users in North America. * Challenges: RUM may require substantial data processing infrastructure to analyze the vast volume of user interaction data in real time. *** **Application Performance Monitoring (APM)** APM focuses on monitoring and optimizing application-level metrics such as response times, throughput, memory consumption, and database query performance. APM tools provide deep visibility into application behavior, often by instrumenting code to measure key metrics. * Benefits: APM is essential for identifying performance bottlenecks, such as slow database queries, inefficient APIs, or memory leaks. * Example Use Case: Diagnosing why a specific API endpoint is causing latency spikes under heavy load. * Challenges: Implementing APM requires careful planning, as excessive instrumentation can add overhead to the application. **Error Tracking and Logging** Error tracking focuses on detecting, logging, and analyzing application errors to help developers diagnose and fix issues effectively. Logs capture details about errors and events, providing valuable context for debugging. * Pitfalls of Defining Errors: * What is an error? The definition varies widely. For some, an error might include user-facing error messages or unhandled exceptions. For others, it might extend to slow-loading assets or deprecated API warnings. * Signaling Theory: Effective error tracking relies on understanding what each sensor (or monitoring system) is intended to capture. Without clear definitions, teams risk alert fatigue or missing critical issues. * Benefits: Centralized logging reduces the time to identify the root cause of an issue, while structured error tracking can prioritize issues affecting user experience. * Example Use Case: Monitoring uncaught exceptions in a JavaScript application and prioritizing fixes for errors impacting 10% of users. * Challenges: Over-logging can lead to excessive noise, making it harder to find actionable insights. **Uptime Monitoring** Uptime monitoring ensures that your web application is available and responsive by periodically sending requests to check its status. Most uptime checks verify that a service responds with a `200 OK` HTTP status. * Benefits: Uptime monitoring offers a straightforward way to track availability, often serving as the first line of defense against outages. * Example Use Case: Monitoring whether an online banking platform’s login page is accessible to users. * Challenges: Simple uptime checks don’t account for partial outages or degraded performance (e.g., slow response times). Or for responses that look okay to a script, but not the users (e.g., the page loads but the only text says ‘server error’). **Server and Infrastructure Monitoring** Server and infrastructure monitoring tracks the health of the hardware, virtual machines, or cloud infrastructure that supports your application. It collects data on CPU usage, memory availability, disk I/O, and network traffic. * Benefits: Essential for ensuring that the underlying infrastructure can meet application demands, especially during peak loads. * Example Use Case: Monitoring CPU usage to detect bottlenecks during a holiday sale on an e-commerce site. * Challenges: Infrastructure monitoring can produce misleading signals in scenarios like network outages that reduce user traffic but increase resource availability, falsely suggesting optimal performance. *** ## Key Metrics to Monitor * Response Time: Time taken by the application to respond to requests. * Error Rate: Frequency of errors in the application. * Throughput: Number of requests processed over a time frame. * Uptime and Availability: Percentage of time the application is operational. ## Benefits of Web Application Monitoring * Instant Downtime Alerts By monitoring your web application directly, you have a better chance of knowing exactly when your service goes down. By alerting on metrics like response time and volume, you can get early indicators of growing problems. * Find trends before they become problems Often, poor performance doesn’t happen all at once. For every user who gets in touch complaining of slow response times, dozens or hundreds will have simply abandoned your service, and written off the quality of your site. Without web application monitoring, you’ll use users for some time before you’re even aware of a problem. ## Limitations of Web Application Monitoring * Dynamic Content Monitoring tools may struggle with rapidly changing, personalized content. A robust testing framework like [Playwright](/docs/detect/synthetic-monitoring/browser-checks/playwright-support) can be helpful for writing smarter assertions about how interfaces *should* look. * Cross-purposes As mentioned in the introduction, Web Application Monitoring has a specific use: finding how well a service performs for users. Once the use case expands into security monitoring or business analytics, mission creep can kill your efficiency. ## Best Practices for Monitoring Web Applications Effective monitoring is essential to maintaining the performance, availability, and security of web applications. However, monitoring is more than just collecting data; it’s a strategic process of learning and acting based on insights. By following these best practices, organizations can ensure their monitoring efforts are both meaningful and actionable. *** * Set Clear Objectives Before implementing monitoring, define what success looks like for your application. Monitoring without clear goals often leads to data overload without actionable insights. * Why It Matters: Objectives guide what data you collect and how you interpret it. For example, minimizing downtime might focus on uptime monitoring and incident alerts, while improving user experience might emphasize performance metrics like load time and responsiveness. * How to Do It: Align your objectives with business goals. For instance, if your goal is to increase conversion rates, focus on monitoring checkout processes and page performance. *** * Choose the Right Monitoring Tools Selecting the right tools for your web application’s unique needs is critical. Not all monitoring tools are created equal, and mismatched tools can lead to unnecessary complexity or blind spots. * Why It Matters: A tool tailored to your application architecture provides more relevant data and reduces noise. * How to Do It: Assess your application stack (e.g., serverless, microservices, or monolithic), the types of metrics you need, and your team’s familiarity with specific tools. For example, use Prometheus for metrics, Loki for logs, and Jaeger for tracing in a Kubernetes-based application. *** * Define Key Performance Indicators (KPIs) KPIs translate business objectives into measurable metrics, bridging the gap between technical monitoring and organizational goals. * Why It Matters: Without KPIs, monitoring efforts can lack focus, leading to wasted resources and misaligned priorities. * How to Do It: Identify KPIs that directly affect user experience or business outcomes, such as uptime, response time, error rates, or user engagement metrics. For example, define a goal like “99.9% uptime for key services over a month.” *** **Monitor User Experience** Modern monitoring goes beyond infrastructure to focus on the end-user journey. Users don’t care if your CPU is underutilized—they care if your site loads quickly and works smoothly. * Why It Matters: User experience (UX) monitoring ensures that technical performance aligns with user satisfaction and retention. * How to Do It: Combine Real User Monitoring (RUM) and Synthetic Monitoring to capture both actual and simulated user interactions. Focus on load times, time to interact (TTI), and error rates that directly impact UX. *** **Implement Continuous Monitoring** Web applications operate in dynamic environments, where issues can arise at any time. Continuous monitoring ensures constant vigilance. * Why It Matters: Continuous monitoring helps teams catch problems early, reducing downtime and improving system reliability. * How to Do It: Automate monitoring across all layers of your stack—servers, APIs, front-end performance, and user interactions. Use tools like CI/CD integrations to monitor deployments for potential issues. *** **Be Proactive with Alerting and Notifications** Alert fatigue is a common problem in monitoring, where too many notifications desensitize teams. A proactive approach focuses on actionable alerts. * Why It Matters: Timely and meaningful alerts enable faster incident resolution while avoiding unnecessary noise. * How to Do It: Configure alerts for critical thresholds and anomalies. For instance, set alerts for unusual spikes in response time or memory usage, but suppress notifications for predictable auto-scaling events. *** **Analyze and Act on Monitoring Data** Data alone is not valuable unless it leads to action. Effective monitoring transforms raw data into insights that drive meaningful improvements. * Why It Matters: Many organizations collect vast amounts of monitoring data but fail to act on it, leaving potential optimizations on the table. * How to Do It: Establish regular review processes to analyze trends, identify recurring issues, and implement fixes. For example, a monthly review of error logs can reveal patterns like frequently failing endpoints. *** **Implement Synthetic Monitoring** Synthetic monitoring simulates user activity to proactively identify potential issues. * Why It Matters: This type of monitoring allows teams to test functionality and performance before users are affected. * How to Do It: Use scripts to mimic common user actions, such as navigating pages, submitting forms, or using APIs. Test critical user paths regularly, especially after updates or deployments. *** **Leverage Real User Monitoring (RUM)** RUM provides insights based on actual user interactions, capturing the diversity of real-world experiences. * Why It Matters: Real user data reflects the performance users experience, including regional differences, device-specific issues, and varying network conditions. * How to Do It: Deploy lightweight tracking scripts to collect metrics such as page load time, interaction speed, and error rates. Segment data by user demographics or device type for targeted improvements. *** **Conduct Regular Performance Audits** Periodic audits ensure that your monitoring strategy remains effective and that your application continues to meet performance expectations. * Why It Matters: Web applications evolve over time, and so do the challenges they face. Regular audits help identify outdated metrics, unnecessary alerts, and new performance bottlenecks. * How to Do It: Schedule audits to review KPIs, monitoring coverage, and tool configurations. For instance, ensure your monitoring setup includes new microservices added to your architecture. *** **Tie Best Practices to Business Goals** Effective monitoring isn’t just about data collection—it’s about using data to improve your application and achieve your organization’s goals. By integrating these best practices into your strategy, you ensure that monitoring becomes a driver of growth, user satisfaction, and operational excellence. ## Top 9 Web Application Monitoring Tools * 1. Datadog Datadog provides a unified platform that integrates metrics, logs, and traces for comprehensive monitoring. It includes features like Real User Monitoring (RUM), Synthetic Monitoring, and Application Performance Monitoring (APM). Personal Experience: When I worked on a large e-commerce platform, Datadog was indispensable for tracking the performance of our microservices. However, the pricing became a pain point as our usage scaled. Integrating some of our custom Prometheus metrics required additional work, and once we were deep into the Datadog ecosystem, migration seemed daunting. It felt like we were trapped in their ecosystem because they handled *everything*, but that also meant learning to live with their constraints. *** * 2. New Relic New Relic is known for its robust APM, RUM, and distributed tracing capabilities. It offers strong support for OpenTelemetry, allowing integration of custom telemetry data. Personal Experience: I worked at New Relic for a number of years, and won’t go into what I learned behind the scenes. As a part of the observability industry now outside of New Relic, I’d say its users have very high standards for how integrated their performance data can be. Small teams and open source projects will never have the slick features of a mature dashboard and monitoring system with hundreds of engineers working to expand and maintain it. *** * 3. Logz.io Logz.io combines the ELK Stack (Elasticsearch, Logstash, and Kibana) with Grafana, providing log management and visualization. It targets engineers familiar with open source tooling but who want the convenience of a managed service. Personal Experience: At a startup, we adopted Logz.io because it fit well with our existing ELK stack workflows. The integration was smooth, and their alerting system helped us catch critical API failures. However, as our log volume grew, the costs escalated, forcing us to revisit whether we should return to managing ELK ourselves. Overall the DIY solution had cheaper infrastructure costs, but way higher team overhead. *** * 4. Sentry Sentry specializes in error tracking, offering detailed stack traces and insights into application crashes. It focuses on developer workflows, making it easy to identify and fix bugs. Personal Experience: We used Sentry extensively during a high-stakes product launch. It was a lifesaver for catching JavaScript errors on our front end and exceptions in our Node.js backend. One memorable instance was tracking down a browser-specific bug affecting a subset of users—Sentry pinpointed the exact issue within minutes. However, the volume of alerts sometimes led to fatigue, requiring us to fine-tune what constituted an actionable error. *** * 5. Icinga Originally a fork of Nagios, Icinga offers monitoring for servers, networks, and applications. It is an open-core product, with its core technology available on GitHub. Personal Experience: This is the one tool on here I’ve never installed or explored, it has a loyal Reddit following and clearly has some knowledge transfer benefits from Nagios. *** * 6. Site24x7 Site24x7 offers application, server, and website monitoring, positioning itself as an all-in-one solution for small-to-medium-sized organizations. Personal Experience: We briefly evaluated Site24x7 for a mid-sized SaaS platform. While the promise of a unified monitoring solution was appealing, the results fell short in every area—synthetic checks, uptime reports, and server monitoring all felt basic compared to more specialized tools. It seemed like Site24x7 was spread too thin trying to cover every use case, and it struggled to meet the expectations of a team with even modest experience in monitoring tools. *** * 7. Raygun Raygun focuses on real user monitoring, crash reporting, and performance tracking. It’s tailored for front-end and mobile developers looking to improve user experience. Personal Experience: We relied on Raygun for monitoring a mobile app with a global user base. It excelled in surfacing crash reports and slow performance patterns tied to specific device types. One particular win was discovering a memory leak issue affecting older Android devices—it gave our team the data we needed to patch the issue quickly. However, integrating Raygun into our CI/CD pipeline required extra effort, as it wasn’t as seamless as some other tools. *** * 8. AppDynamics AppDynamics, now part of Cisco, provides APM capabilities with a focus on deep-dive analytics and business transaction monitoring. Personal Experience: At an enterprise client, we implemented AppDynamics to monitor a sprawling microservices architecture. Its transaction-based insights helped us identify bottlenecks in our checkout flow, translating directly into improved conversion rates. However, navigating its interface felt cumbersome, and despite its advanced features, team adoption was slower than anticipated due to the tool’s complexity. It became a love-hate relationship—powerful but not always user-friendly. *** * 9. IBM Instana Instana promises AI-driven insights and automatic monitoring for dynamic applications. Its focus is on reducing manual configuration by detecting and mapping dependencies in real time. Personal Experience: During an initiative to modernize a legacy application, Instana was pitched to us as the “next-generation AI observability solution.” While it delivered on automated dependency mapping, the AI insights often felt like little more than a repackaging of existing metrics with an added buzzword. It was helpful for identifying sudden resource spikes, but I often found myself relying on other tools for detailed root-cause analysis. The promises of “AI-driven” observability didn’t live up to the hype. ## Open Source Monitoring: a key part of web application monitoring Open source monitoring plays a pivotal role in modern cloud-native web application monitoring by leveraging community-driven tools to provide robust, scalable, and accessible monitoring solutions. These tools empower organizations to manage the complexity of distributed systems without being locked into proprietary solutions. **Challenges in Cloud-Native Observability** Cloud-native applications, by design, introduce new complexities: * Obfuscation: Dependencies across microservices, Kubernetes orchestration, and cloud-managed services obscure system behavior. * Dynamic Dependencies: The interactions among thousands of microservices, infrastructure layers, and APIs shift dynamically with scaling and updates. * Data Volume: High data granularity across logs, metrics, traces, and flows creates immense operational overhead to derive actionable insights. **Core Components of Open Source Monitoring** 1. Metrics Collection with Prometheus Prometheus is the cornerstone of open source monitoring, providing time-series data collection and querying capabilities. With exporters like Node Exporter and Kubernetes State Metrics (KSM), it gathers metrics at both the system and application levels. 2. Log Aggregation with Loki Loki collects and queries logs efficiently, ensuring contextual insights alongside metrics. Integrated with Prometheus and Kubernetes, Loki enables rapid troubleshooting. 3. Distributed Tracing with Jaeger Jaeger offers a standardized approach to tracing requests across microservices, enabling visibility into service-to-service interactions and latency. 4. Kernel-Level Observability with eBPF The extended Berkeley Packet Filter (eBPF) collects real-time data on network flows and application behavior, bypassing traditional agents and minimizing performance overhead. Open source monitoring is not just a viable alternative to proprietary solutions—it is the backbone of modern observability. With tools like Prometheus, Loki, Jaeger, and eBPF, organizations can effectively monitor cloud-native applications, dynamically adjust to changing workloads, and achieve operational excellence. By embracing these technologies, teams can focus on delivering exceptional user experiences while keeping operational costs in check. ## How Can Checkly Help with Web Application Monitoring [Checkly](https://www.checklyhq.com/) offers an integrated platform for synthetic monitoring and API testing, ensuring applications are functional and performant. For simulating user requests to perform synthetic monitoring, Checkly harnesses the power of the open source automation tool Playwright. With [Checkly and Playwright](/docs/detect/synthetic-monitoring/browser-checks/playwright-support), there’s no user behavior that you can’t test, and no critical path you can’t monitor continually. ## Conclusion Web application monitoring is essential for ensuring high performance, user satisfaction, and business success. With various tools and techniques available, businesses can proactively manage their applications and deliver exceptional user experiences. # What is TCP? - how it works and why it matters to application developers Source: https://www.checklyhq.com/docs/learn/monitoring/what-is-tcp Explore web application monitoring to boost performance and reliability with real user insights, performance tracking, and top tools. As an application developer, you might think that Transmission Control Protocol (TCP) is something only network engineers need to worry about. However, as application developers take on more of the operational design and running of their applications, these previously 'low level' protocols come to the fore. Understanding the basics of TCP can help you build better, more reliable applications. This document will explain what TCP is, how it works, and what you need to know as an application developer. ## What is TCP? TCP stands for Transmission Control Protocol. It is one of the main protocols used in the Internet Protocol (IP) suite. TCP is responsible for ensuring that data sent from one device to another arrives correctly and in the right order. TCP connections underlie the application-layer connections like HTTP connections that we're more familiar with as app developers. It is a connection-oriented protocol, which means it establishes a connection between the sender and receiver before data is sent. ## Why Should Application Developers Care About TCP? Even though TCP operates at a lower level than your application code, it directly impacts how your application performs. Here are a few reasons why you should care about TCP: 1. **Non-http services**: TCP is used (without any HTTP layer) by databases, message queues, and mail servers, all of which your application may rely on to deliver the best customer experience. Understanding how TCP works will help you debug problems integrating these dependencies. 2. **TCP is part of flow control**: TCP manages the rate at which data is sent to prevent overwhelming the receiver, by understanding these components, you protect your application from getting rate limited and experiencing edge case failures 3. **Connection management**: TCP handles the establishment, maintenance, and termination of connections between devices, and with dynamic services like user-defined containers you may need to manage these connections more than you would expect! Understanding these aspects can help you optimize your application's performance and troubleshoot issues more effectively. ## How TCP Works ### TCP Connection Establishment Before data can be sent, a TCP connection must be established between the sender (client) and the receiver (server). This process is known as the **three-way handshake**: A TCP Handshake

1. **SYN**: The client sends a SYN (synchronize) packet to the server to initiate a connection. 2. **SYN-ACK**: The server responds with a SYN-ACK (synchronize-acknowledge) packet to acknowledge the request. 3. **ACK**: The client sends an ACK (acknowledge) packet back to the server to confirm the connection. Once the three-way handshake is complete, the connection is established, and data can be transmitted. ### Data Transmission After the connection is established, data is sent in segments. Each segment contains a sequence number that helps the receiver reassemble the data in the correct order. The receiver sends an acknowledgment (ACK) for each segment received. If the sender does not receive an ACK within a certain time, it will retransmit the segment. ### Flow Control TCP uses a mechanism called **flow control** to ensure that the sender does not overwhelm the receiver with too much data. The receiver advertises a window size, which indicates how much data it can accept. The sender adjusts the amount of data it sends based on this window size. ### Congestion Control TCP also implements **congestion control** to prevent network congestion. If the network is congested, TCP will reduce the rate at which it sends data. This is done using algorithms like **Slow Start**, **Congestion Avoidance**, and **Fast Retransmit**. ### Connection Termination When the data transfer is complete, the TCP connection is terminated using a four-step process: 1. **FIN**: The client sends a FIN (finish) packet to the server to indicate that it wants to close the connection. 2. **ACK**: The server sends an ACK packet to acknowledge the FIN packet. 3. **FIN**: The server sends its own FIN packet to the client. 4. **ACK**: The client sends an ACK packet to acknowledge the server's FIN packet. After this process, the connection is closed. ## Key TCP Concepts for Application Developers ### Ports and Sockets In TCP, communication is identified by a pair of **sockets**. A socket is a combination of an IP address and a port number. The IP address identifies the device, and the port number identifies the specific application or service on that device. As an application developer, you need to be aware of the ports your application uses. Common services use well-known ports (e.g., HTTP uses port 80, HTTPS uses port 443), but your application may use different ports. ### Buffering TCP uses buffers to store data before it is sent or after it is received. As a developer, you can control the size of these buffers, which can impact performance. Larger buffers can improve throughput but may increase latenc. ### Keep-Alive TCP connections can be kept alive even when no data is being transmitted. This is useful for applications that need to maintain a connection for a long time. However, keep-alive packets can consume network resources, so you should use them judiciously. ### Timeouts TCP uses timeouts to detect when a connection is no longer active. If no data is received within a certain time, the connection may be closed. As a developer, you can configure these timeouts to suit your application's needs. ### Error Handling TCP handles most errors automatically, but you should still be aware of potential issues. For example, if a connection is lost, TCP will attempt to retransmit data, but if the connection cannot be reestablished, your application may need to handle the error. ## TCP and Application Performance ### Latency Latency is the time it takes for data to travel from the sender to the receiver. TCP introduces some latency due to the three-way handshake, acknowledgments, and congestion control. Understanding how TCP affects latency can help you optimize your application's performance. ### Throughput Throughput is the amount of data that can be transmitted in a given time. TCP's flow and congestion control mechanisms can impact throughput. By tuning buffer sizes and window sizes, you can improve your application's throughput. ### Scalability TCP connections consume resources on both the client and server. If your application needs to handle many simultaneous connections, you should consider how TCP's connection management affects scalability. Techniques like connection pooling and load balancing can help. ## Common TCP Issues and How to Troubleshoot Them ### Packet Loss Packet loss can occur due to network congestion or hardware issues. TCP will retransmit lost packets, but frequent packet loss can degrade performance. Use tools like `ping` or `traceroute` to diagnose network issues. ### Slow Performance Slow performance can be caused by many factors, including high latency, low throughput, or network congestion. Use tools like `netstat` or `tcpdump` to analyze TCP connections and identify bottlenecks. ## Best Practices for Application Developers ### Use Persistent Connections Persistent connections reduce the overhead of establishing and tearing down TCP connections. This can improve performance, especially for applications that make many small requests. ### Optimize Buffer Sizes Adjusting the size of TCP buffers can improve performance. Larger buffers can increase throughput, while smaller buffers can reduce latency. Experiment with different buffer sizes to find the optimal configuration for your application. ### Monitor TCP Metrics Monitoring TCP metrics like latency, throughput, and packet loss can help you identify performance issues. Use monitoring tools to track these metrics and make adjustments as needed. ### Handle Errors Gracefully Even though TCP handles most errors automatically, your application should still handle errors gracefully. For example, if a connection is lost, your application should attempt to reconnect or notify the user. ### Test Under Realistic Conditions Test your application under realistic network conditions to ensure it performs well. Use tools that simulate network latency, packet loss, and congestion to identify potential issues. Essentially, once you're trying to test the integration between multiple dependencies connecting via multiple protocols, consider testing in a production environment with [synthetic monitoring](https://www.checklyhq.com/blog/what-is-synthetic-monitoring/) rather than trying to simulate the scenario with mocks. ## TCP and HTTP As mentioned above HTTP connections are application-level connections with TCP handling the network-layer connection. A few key concepts for those familiar with HTTP: ### 1. **Persistent Connections (HTTP Keep-Alive)** * By default, HTTP/1.1 uses **persistent connections**, meaning the same TCP connection can be reused for multiple HTTP requests and responses. * This reduces the overhead of establishing and closing TCP connections, improving performance. * The `Connection: keep-alive` header is used to enable persistent connections. ### 2. **Connection Pooling** * To further optimize performance, applications often use [**connection pooling**](https://node-postgres.com/features/pooling), where a pool of TCP connections is maintained and reused for multiple HTTP requests. * This is especially useful for applications that make many HTTP requests to the same server (e.g., APIs, microservices). ### 3. **Latency and the Three-Way Handshake** * The TCP three-way handshake introduces latency, especially for short-lived HTTP requests. * Techniques like **HTTP/2 multiplexing** and **HTTP/3** (which uses QUIC instead of TCP) can reduce this latency. * Applications using many short lived requests (e.g. streaming a game to a browser) will want to switch from HTTP/1 connections. ### 4. **Error Handling** * TCP handles most network-level errors (e.g., packet loss, retransmission), but your application should still handle HTTP-level errors (e.g., `4xx` and `5xx` status codes). ### 5. **SSL/TLS and HTTPS** * When using HTTPS, the SSL/TLS handshake occurs after the TCP connection is established but before the HTTP request is sent. * This adds additional latency but ensures the connection is secure. ### 6. How HTTP/2 and HTTP/3 Change the Relationship **HTTP/2** * HTTP/2 introduces **multiplexing**, allowing multiple HTTP requests and responses to be sent concurrently over a single TCP connection. * This reduces the need for multiple TCP connections and improves performance. * However, HTTP/2 still relies on TCP, so it inherits TCP’s limitations (e.g., head-of-line blocking). **HTTP/3** * HTTP/3 replaces TCP with **QUIC**, a transport protocol built on UDP. * QUIC eliminates the three-way handshake, reduces latency, and handles packet loss more efficiently. * This makes HTTP/3 faster and more resilient, especially in high-latency or lossy networks. * Currently adoption of HTTP/3 is [growing with almost 30% of the market supporting](https://w3techs.com/technologies/details/ce-http3) HTTP/3 connections. ## Conclusion While TCP is often considered the domain of network engineers, understanding its basics is crucial for application developers. By understanding how TCP works, you can optimize your application's performance, troubleshoot issues more effectively, and build more reliable applications. Remember to monitor TCP metrics, optimize buffer sizes, and handle errors gracefully to ensure your application performs well under all conditions. ## Additional Resources * [**RFC 793**](https://www.rfc-editor.org/info/rfc793): The official specification for TCP. * [**Wireshark**](https://www.wireshark.org/): A network protocol analyzer that can help you inspect TCP traffic. * [**TCP/IP Illustrated, Volume 1**](http://www.kohala.com/start/tcpipiv1.html): A comprehensive book on TCP/IP protocols. By following the best practices outlined in this document, you can ensure that your application leverages TCP effectively to deliver a reliable and high-performance user experience. # Fundamentals of Context Propagation in OpenTelemetry Source: https://www.checklyhq.com/docs/learn/opentelemetry/context-propagation One of the most basic features of almost any OpenTelemetry tool, is support for context propagation, a fundamental component that sends information from one service to another and allows traces to cover multiple services. Context propagation turns separate service logs into complete traces. It's what makes modern observability possible. Why does OpenTelemetry exist? It grew out of the OpenTracing project, which itself was a response to a basic problem of modern architectures: while microservices had greatly improved operational efficiencey and developer velocity, microservice architectures had proved a disaster for observability. Imagine an architecture like Uber’s: A diagram of Uber’s Microservices, from a paper by Muhammad Ahsan (CC 4.0 Attribution License)

A diagram of Uber’s Microservices, from a paper by Muhammad Ahsan (CC 4.0 Attribution License)

*A diagram of Uber’s thousands of Microservices* Imagine the impossibility of diagnosing problems found during end-to-end testing, if the only tracing or observability we have is isolated to a single service’s context. And so the OpenTelemetry project was founded, fundamentally, to provide insights that go across multiple services within a single cloud. In pursuit of this lofty goal, one of the most basic features of almost any OpenTelemetry tool, is support for context propagation, a fundamental component that sends information from one service to another and allows traces to cover multiple services. ### What is Context? In OpenTelemetry, context is the information that flows through your system with each request. It carries important details like: * Trace IDs (to connect all operations in a request) * Baggage (custom key-value pairs you want to pass along) * Other metadata needed for observability Context standards are not defined by the OpenTelemetry project but rather are baced on the [W3C Trace Context](https://www.w3.org/TR/trace-context/) standards. ### Why Context Matters As mentioned above, modern architectures are complicated. Many outages and other incidents end with engineers examining the logs from a single service, but to get real insight into how requests are processed by your system as a whole, you need context. Without context: * Logs, traces, and metrics can't be connected * You lose visibility in distributed systems * Debugging becomes much harder ### How Propagation Works 1. **Injection**: When a request starts, the context (trace ID, etc.) gets added to headers 2. **Extraction**: When another service receives the request, it pulls the context from headers 3. **Passing Along**: The context continues flowing through all services in the request Note that while most OpenTelemetry SDKs and tools propagate context automatically, there’s also the [propagation API](https://opentelemetry.io/docs/specs/otel/context/api-propagators/) if you need to manually propagate context. For example, if you are able to see that two time spans should be part of one trace via an unusual method like examining the messages body with a regex, you’d want to work with the propagation API. ### The Standards Behind It OpenTelemetry uses the W3C Trace Context standard (traceparent header) for compatibility. This means: * Works across different languages and frameworks * Supported by many observability tools * Ensures consistent behavior everywhere ### Key Components * **Context API**: Stores and manages the current context * **Propagators**: Handle injecting/extracting context (usually from HTTP headers) * **Baggage**: Lets you add custom data that travels with the trace Baggage is worth a pointer since it’s important to know that you can add operational information beyond observability data, and thereby piggyback off the great work done for context propagation within the OTel project. This might be used to do something like marking test requests sent by [synthetic monitoring](https://www.checklyhq.com/docs/), so that your datastore knows to route these requests to a test DB. ### In Practice When Service A calls Service B: 1. Service A adds trace context to the request headers 2. Service B reads those headers to continue the trace 3. All spans from both services connect automatically ### Why This is Powerful * No manual correlation needed * Works across any protocol (HTTP, gRPC, queues, etc.) * Enables complete observability of distributed transactions Context propagation turns separate operations into connected stories - making complex systems much easier to understand and troubleshoot. ## How Context Propagation Unlocks Efficient Monitoring: Checkly Traces At Checkly, we had a problem. We wanted to let our users connect data on the backend performance of their application when handling requests sent by Checkly as part of synthetic user monitoring. For example, if a single postgres query was taking a long time and slowing down one type of user request, we wanted to see what request had taken so long. But we didn’t want to send large amounts of unnecessary data, using up user’s bandwidth and resources. To continue the analogy, we only wanted to send the postgres query data for our specific test request, not all postgres query performance from everywhere. With OpenTelemetry context propagation, this proved feature was easy to implement, and performs well. ### How it works: Context Propagation and Checkly Traces 1. A user configures their Checkly test service to send requests with an additional header, marking this request as coming from Checkly 2. The user adds a few lines of configuration to their OTel Collector, to send on relevant traces 3. The service, instrumented with OpenTelemetry libraries, sees the header and passes it on between services 4. When reporting a trace to the OpenTelemetry Collector, if the trace has the Checkly header, it’s sent on to Checkly 5. Checkly’s reports on check results now have backend data, having sent only relevant traces to the Checkly server A checkly dashboard

*Users of checkly data can see relevant backend performance data, but don’t send any timespans that aren’t necessary.* This is just one example of how context propagation is a key superpower for OpenTelemetry. ## Conclusions: OpenTelemetry Context Propagation is Fundamental to Observability Context propagation turns separate service logs into complete traces. It's what makes modern observability possible. As systems grow more complex, this simple way of connecting services becomes essential for: * Faster debugging * Better performance tuning * Understanding real user experiences OpenTelemetry's implementation gives you this power without requiring changes to how your services communicate. That's why it's becoming the standard way to observe distributed systems. # Getting Started with Observability Source: https://www.checklyhq.com/docs/learn/opentelemetry/getting-started-with-observability Learn OpenTelemetry with Checkly. Add monitoring to every piece of your stack with the open standards and open-source tools. ## A beginner's guide **Observability** is the practice of understanding the internal state of a system by examining the outputs it generates—such as logs, metrics, and traces. OpenTelemetry (OTel) plays a key role in modern observability by offering open standards for instrumenting code, gathering telemetry data, and managing this data through centralized collectors. ### Why Observability Matters In systems that adopt **microservices** architecture, tracking system health becomes challenging. Unlike monolithic systems, where a few experts can oversee the whole system, microservices distribute responsibilities across many independent services. This fragmentation makes it difficult to pinpoint issues and monitor end-to-end system behavior. Observability addresses these gaps by enabling better monitoring and faster resolution of incidents. At the same time that microservice architectures cry out for better observability, the fragmentation of these code execution environments means that tracing a request through this stack has gotten exponentially more difficult. From these needs was OpenTelemetry born (OpenTelemetry was born by merging the OpenCensus and OpenTracing projects). ## What OpenTelemetry is, and isn't As you browse the many instrumentation tools and [collector](/docs/learn/opentelemetry/what-is-the-otel-collector/) modules included in the OpenTelemetry project, it can seem like the project is incredibly vast. However there are some clear demarcations around what is and isn't part of the project. ### The OpenTelemetry project includes The Otel project includes everything you need to collect data about your system, with standards in place to make sure later analysis is standardized. This covers: * A standard for internal observability APIs, and a language standard for any OpenTelemetry SDKs * A standard protocol for sending data - the OpenTelemetry transfer protocol or OTLP * Libraries to instrument applications, supporting all common web development languages * The OpenTelemetry collector, an all-purpose data forwarding and filtering tool for controlling OTel data * Various other community modules, including plugins for the above systems, and data gathering tools for Kubernetes, FaaS, and others. ### What isn't part of OpenTelemetry? Often you'll see a question posted to the effect of 'how can the OpenTelemetry collector make me a dashboard?' Someone asking this question has gotten confused: while the collector receives data, it only does so to immediately forward the data somewhere else. The OpenTelemetry project is vendor-agnostic, and as such it doesn't compete with any open source or SaaS observability dashboard service. This helps explain why the Otel project does **not** include: * Data storage * Data querying * Dashboards, charting, and other components for viewing data * Alerting and monitoring tools Often in these guides, we'll mention open source tools like Prometheus and Grafana for storing and displaying OpenTelemetry data. However OpenTelemetry is not preferential to these tools, and with a unified standard for transmitting data, any vendor that supports the OTLP standard should work to receive OpenTelemetry data. ### The Three Pillars of Observability OpenTelemetry enables observability through three core data types: 1. **Metrics**: * Numerical summaries of system behavior (e.g., CPU usage, request counts). * Provide high-level insights into trends and overall performance. * Metrics are efficient to collect and store, making them suitable for monitoring at scale. 2. **Logs**: * Detailed records of events or states within a system. * Offer a complete picture of system operations but can become unwieldy in large volumes. * While useful for post-mortem analysis, starting with logs during a live incident may slow down troubleshooting. 3. **Traces**: * Capture the lifecycle of a request as it moves through various services in a system. * Tracing helps identify the components involved in a request and their performance (e.g., through waterfall charts). * Distributed tracing extends this concept to microservices, ensuring that spans from different services are correlated correctly. ### Distributed Tracing with OpenTelemetry Distributed tracing relies on propagating a **trace context** across services. Each service contributes spans to the trace, which are visualized in sequence to understand the request's journey. OpenTelemetry makes this possible by defining trace headers that are passed across service boundaries. ### Getting Started with OpenTelemetry To begin using OpenTelemetry, you will follow three general steps: * Start generating OTel data, generally by instrumenting your application code. * Report your data to some location, ether with the OpenTelemetry collector, or by directly reporting data to an OTLP endpoint somewhere. * Implement storage and display of your data (as mentioned above, this step is not part of the OTel project and multiple solutions exist). The flexibility of the collector enables smooth transitions between direct reporting and more complex data pipelines as your observability needs grow. ### Relevant Resources on OpenTelemetry * **Metrics Overview**: [Learn how OpenTelemetry handles metrics to provide high-level insights into your system's performance](/docs/learn/opentelemetry/metrics). * **Logging with OpenTelemetry**: [Discover how OpenTelemetry integrates with existing logging libraries and enhances log data correlation across microservices](https://opentelemetry.io/docs/specs/otel/logs/). * **Quick Start Guide**: A guide for setting up OpenTelemetry quickly to start monitoring your applications [here](https://opentelemetry.io/docs/getting-started/). These resources explain the core pillars of observability, as well as how to use OpenTelemetry’s **Collector** to manage and export telemetry data to observability platforms like Prometheus and Grafana. ## Conclusion Observability with OpenTelemetry empowers teams to quickly detect, understand, and resolve issues in microservices environments. By adopting this open framework, organizations gain the ability to monitor complex systems without relying on proprietary tools, ensuring scalability and interoperability across platforms. By implementing OpenTelemetry, you gain a unified view of system health across microservices and can ensure your monitoring solutions are scalable and vendor-neutral. # How to instrument your application with OpenTelemetry Source: https://www.checklyhq.com/docs/learn/opentelemetry/how-to-instrument OpenTelemetry offers two main strategies for adding observability to your application - direct instrumentation and auto-instrumentation. Each method serves specific use cases, allowing developers to capture telemetry data efficiently while balancing flexibility and simplicity. OpenTelemetry offers two main strategies for adding observability to your application, learn how they work together. OpenTelemetry offers two main strategies for adding observability to your application: **direct instrumentation** and **auto-instrumentation**. Each method serves specific use cases, allowing developers to capture telemetry data efficiently while balancing flexibility and simplicity. *** ### Direct Instrumentation Direct instrumentation involves **manually adding code** to your application to generate telemetry data. This requires developers to use the OpenTelemetry SDK specific to their programming language, explicitly invoking functions to create traces, metrics, or logs. * **Example Use Case**: You might define spans that wrap critical sections of code to monitor database queries or key service calls. * **Advantages**: 1. **Precise Control**: Developers control which parts of the application generate telemetry data. 2. **Contextual Richness**: Since code is written by hand, you can include detailed metadata relevant to your business logic. 3. **Customization**: Allows for intricate tracking of custom events, metrics, and logs that fit your specific needs. This approach ensures deeper insights but requires more engineering effort upfront and ongoing maintenance to ensure proper coverage and updates as the code evolves. There are of course 1,001 ways that a clever developer could ensure that the intervention needed even for direct instrumentation was miniman. For example, by wrapping the function that first receives a request, and the function class to ensure that future function calls are reported to a tracer. Direct instrumentation doesn't mean modifying n+1 lines of code, but it does require a fair amount of effort to get launched the first time, no matter how clever your developers are. ### Auto-Instrumentation Auto-instrumentation eliminates the need for manual code changes by **injecting telemetry at runtime**. This is often achieved through agents or middleware that modify existing code dynamically. For example, Java or Python applications can use agents to hook into libraries and frameworks they rely on. * **Example Use Case**: A web server library automatically generates spans for HTTP requests without developer involvement. * **Advantages**: 1. **Fast Setup**: No changes to the codebase are required, making it easy to get started. 2. **Coverage for Standard Libraries**: Many popular libraries and frameworks come with built-in support for auto-instrumentation. 3. **Consistency**: Ensures consistent telemetry for common operations, reducing the risk of missing critical observability data. Auto-instrumentation may not offer the same level of granularity or customization as manual instrumentation. Automatic instrumentation can produce unnecessarily ‘noisy’ instrumentation, as just one example the NodeJS auto-instrumentation can produce traces with [hundreds of file system spans](https://github.com/open-telemetry/opentelemetry-js-contrib/issues/1344) that can even overrun available memory. Finally, auto-instrumentation won’t generally produce useful results if something really unusual is happening inside your application. At base, automatic instrumentation assumes that your service is some kind of online application handling requests and sending responses. If that’s not the case, for example if your application is a machine learning system processing through a database and taking over 48 hours to produce results, you may get less useful information. In these cases, you may want to look into manually reporting [OpenTelemetry metrics](/docs/learn/opentelemetry/metrics). ### When to Use Each * **Direct instrumentation** is ideal for teams seeking maximum flexibility, particularly for custom workflows or applications with unique behavior. * **Auto-instrumentation** is better suited for scenarios where rapid observability is needed, or when standard libraries provide sufficient coverage. In many cases, these methods can complement each other. Start with auto-instrumentation to gain quick insights, then enhance with direct instrumentation to capture application-specific details. For more on how to add instrumentation to specifically integrate with Checkly Traces, [see our Traces documentation](/docs/resolve/traces/overview). # Logging in OpenTelemetry, an Engineer’s Guide Source: https://www.checklyhq.com/docs/learn/opentelemetry/logs This article will go over the basics, and answer common questions for those getting started with logging in OpenTelemetry. OpenTelemetry elevates logging from isolated text snippets to fully contextualized observability data that integrates seamlessly with traces and metrics. Logging is the one of the oldest signals we get from our production software, but with the emphasis on tracing between services, logging is often not the first feature we think about with OpenTelemetry. However, logging can be a key part of any modern observability toolkit, and this article will go over the basics, and answer common questions for those getting started. ## Why isn’t logging fully supported in so many OTel SDKs? Taking a look at the ‘Language APIs and SDKs’ section of the OpenTelemetry documentation, and you’ll discover something odd or even concerning, many languages don’t have full logging support! A table of OpenTelemetry Frameworks

*A table of language-framework support for OpenTelemetry Signals, retrieved in March 2025* In fact only Java, C, and PHP have ‘stable’ logs support. Is this a cause for concern? In a word ‘no’, this table covers whether, when using the OpenTelemetry SDK, you can make a call to this SDK to emit logging. It is true that in many SDKs this isn’t yet fully supported, but this shouldn’t be a great concern to you because **you are already emitting logging just fine** in your production environment. There are lots of ways to add logging to the signals being handled by the OpenTelemetry collector and your OpenTelemetry datastore, which hook in to your current logging streams. ## OpenTelemetry and Structured Logging When you send logs from traces or events, you could just write raw text. But raw logs are hard to read and parse later. OpenTelemetry prefers **structured logging** instead. Structured logging is a best practice recommended by AWS and other cloud providers as well, since even a simple search of past logs can become prohibitively expensive if you’re ### What is Structured Logging? Structured logging means logs are saved in a clear, machine-readable format—like JSON—instead of plain text. Each log has: * **Fields** (key-value pairs) for details like timestamps, severity, and context. * **Consistent formatting**, making logs easier to search, filter, and analyze. ### Why OpenTelemetry Uses Structured Logs 1. **Better for Machines** – Tools can automatically process logs without messy parsing. 2. **Connects to Traces** – Logs can link to traces and spans, helping debug issues faster. 3. **Standardized** – OTel defines a common schema so logs work across different systems. ### How OTel Handles Logs * **Log Records** include: * Timestamp * Severity (e.g., ERROR, INFO) * Message body * Attributes (extra context like `user_id`) * **Integration** with traces and metrics for full observability. ### Example (Structured vs. Raw) **Raw (Hard to Parse):** `"Error: User login failed"` **Structured (Easy to Use):** ```json theme={null} { "timestamp": "2024-05-21T10:00:00Z", "severity": "ERROR", "body": "User login failed", "attributes": { "user_id": "123", "error_code": "AUTH_401" } } ``` Structured logs save time and make debugging easier. OpenTelemetry builds this into its design, so your logs work well with other observability tools. ### Even in logging, cardinality explosion is possible When asking for advice from a senior engineer on labelling new entries into logging with structured labels, and deriving metrics from logs, she replied: If you turn high-cardinality fields (like `user_ip`) into metric labels, you’ll create thousands of unique time series. **💥 Boom—metrics get huge, slow, and expensive!** ✅ **Structured logs are safer**—you don’t have to predefine every possible label combo. Just extract metrics *after* filtering/aggregating logs (e.g., count errors by `error_type`, not by `user_id`). 🔥 **Why?** Every unique label combination: * Becomes a separate metric line 📈 * Lives in memory (your app + Prometheus) �� * Slows down scrapes & garbage collection 🐌 💡 **Pro tip:** Use logs for granular data, metrics for aggregates. Pre-aggregate before exporting! Logging, then, should be seen as similar to event data, and be the storehouse for aggregate, specific information like IP addresses and specific page routes. ## The Promise of OpenTelemetry Logging: Full Context on Every Line ### The Problem with Traditional Logging Today, logs, traces, and metrics live in separate worlds. They use different tools, formats, and collection methods, making it hard to connect them. * **Weak links between signals** – Logs might have timestamps or vague tags, but they don’t *truly* connect to traces or metrics. * **No shared context** – In distributed systems, logs from different services don’t automatically tie together. * **Manual, messy correlation** – You often stitch things together using timestamps or guesswork. This leads to **fragile observability**—logs tell part of the story, but not the whole picture. ### OpenTelemetry’s Solution: Built-in Context OpenTelemetry (OTel) fixes this by **propagating full context in every log line**. * **Traces + logs = best friends** – Each log carries trace and span IDs, so you can jump from a log to the exact trace (and vice versa). * **Consistent metadata** – Kubernetes pod info, service names, and other attributes are added *automatically* by the OTel Collector—no manual tagging needed. * **Distributed systems just work** – Logs from different services share the same context, so you can follow a request across your entire system. ### How It Works 1. **Structured logs** – Logs follow OTel’s data model (like JSON with clear fields). 2. **Automatic enrichment** – The OTel Collector adds context (e.g., `k8s.pod.name`) to logs, traces, and metrics *the same way*. 3. **No re-engineering of existing libraries** – Works with existing logging libraries (Log4j, Logback, etc.) via bridges. ### Why This Matters * **No more guessing** – Every log line knows exactly where it came from and how it fits into the bigger picture. * **Better debugging** – Jump from a log error directly to the trace, metrics, and related logs—no manual searching. * **Future-proof** – Even legacy apps can join the OTel ecosystem via the Collector. ### The Promise, not yet fulfilled OpenTelemetry promises to turn logs from isolated snippets into **fully connected observability data**. As noted above, not every language SDK has full logging support, meaning this isn’t completely possible now. And the addition of enrichment data isn’t an easy or automatic process at the collector level (the collector is stateless and can only enrich logs based on what it’s processing at that moment). However, logging with OpenTelemetry has a bright future, and adopting now means you’re likely to be producing highly valuable logs in the next few years that prove worth the effort. ## The Collector: OpenTelemetry’s Super-Powered Logging Proxy ### The Problem with Legacy Log Collection Most apps today log to files or stdout—simple, but messy. To get those logs into observability tools, you need: * **Parsers** (for JSON, CSV, plain text, etc.) * **Log tailing** (handling rotation, partial reads) * **Enrichment** (adding metadata like `trace_id`) This is fragile work. A typo in a log line can break parsing. Correlating logs with traces? Often manual. ### Enter the OpenTelemetry Collector 🦸 The Collector acts as a **universal log processor**. It can: 1. **Read logs from files/stdout** – No app changes needed. 2. **Auto-add context** – Injects `trace_id`, Kubernetes labels, and more. 3. **Convert to OTLP** – Sends logs in OpenTelemetry’s clean, structured format. **Bonus:** It does the same for traces and metrics—enriching all signals consistently. ### Two Ways to Use It ### Option 1: File/Stdout → Collector (No Code Changes) * **How it works**: The Collector tails log files, parses them, and adds context. * **Good for**: Legacy apps, third-party software. * **Tradeoff**: Parsing can be tricky if log formats are inconsistent. ### Option 2: App → Collector (Direct OTLP) * **How it works**: Use an OTel "log appender" (e.g., for Log4j) to send logs directly to the Collector. * **Good for**: New apps or those you control. * **Why?** No file parsing, no lost logs on crashes, full structured data. ### The Best Part: Zero Manual Enrichment No more editing every `log.info()` call! The Collector handles: * **Trace context** (auto-injects `trace_id`/`span_id`) * **Resource attributes** (like `k8s.pod.name`) * **Routing** (send logs to Loki, Splunk, etc.) ### Why This Wins * **No more "log soup"** – Every log line has rich, queryable fields. * **Correlation just works** – Logs link to traces automatically. * **One tool to rule them all** – Same Collector works for logs, traces, and metrics. How to get started? Start with file collection, then phase in direct OTLP for critical apps. ## Conclusions: OpenTelemetry is Critical to Logging Logging has always been a fundamental part of understanding software behavior, but in modern distributed systems, traditional logging approaches fall short. OpenTelemetry elevates logging from isolated text snippets to fully contextualized observability data that integrates seamlessly with traces and metrics. The power of OpenTelemetry logging lies in its structured approach. Rather than dealing with ambiguous plain text logs, systems can emit rich, machine-readable records with consistent timestamps, severity levels, and attributes. This structure enables more reliable parsing, better searching, and most importantly, automatic correlation with trace data. When every log carries trace context, debugging distributed transactions becomes dramatically simpler. While it's true that not all language SDKs currently offer full logging support, this gap matters less than one might think. The OpenTelemetry Collector provides a powerful bridge, capable of processing existing log files and enriching them with contextual metadata. Whether through direct instrumentation or collector-based processing, teams can start benefiting from better log integration today. The Collector deserves special attention as the workhorse of OpenTelemetry logging. It handles the messy realities of log collection - file tailing, parsing various formats, and log rotation - while adding valuable context like trace IDs and Kubernetes metadata. This happens automatically, without requiring changes to application code. For organizations running diverse systems, this represents a practical path to better observability. Looking ahead, OpenTelemetry's vision for logging is clear: logs should be first-class citizens in observability, not isolated artifacts. As the ecosystem matures, we can expect tighter integration between logging SDKs and collectors, making context-rich logging the default rather than the exception. The transition to OpenTelemetry logging requires some investment, but the payoff is substantial. Teams that adopt this approach will find themselves spending less time hunting through logs and more time solving real problems. In an era of complex distributed systems, this level of observability isn't just nice to have - it's essential for maintaining reliable services and efficient debugging workflows. For engineering organizations serious about observability, embracing OpenTelemetry's approach to logging isn't an optional upgrade - it's a critical evolution of one of our most fundamental debugging tools. The future of logging is contextual, structured, and integrated, and that future is being built today in OpenTelemetry. # What are OpenTelemetry Metrics? Source: https://www.checklyhq.com/docs/learn/opentelemetry/metrics OpenTelemetry Metrics play a critical role in monitoring applications by offering a way to capture and analyze key metrics in a standardized, scalable manner. Whether you're managing a complex microservices architecture or a simpler system, OpenTelemetry helps track essential statistics that reveal the health and performance of your services. Learn more about OpenTelemetry & Monitoring with Checkly. Explore metrics, one of the three pillars of observability. **OpenTelemetry Metrics** play a critical role in monitoring applications by offering a way to capture and analyze key metrics in a standardized, scalable manner. Whether you're managing a complex microservices architecture or a simpler system, OpenTelemetry helps track essential statistics that reveal the health and performance of your services. *** ## What are Metrics? Metrics represent **quantitative measurements** of your system’s health and behavior. They provide insights into performance trends, such as: * **CPU usage** over time * **Request rates** per endpoint * **Error counts** or failure rates * **Latency** in handling requests Metrics are lightweight and highly efficient to collect, aggregate, and query. They help identify patterns and anomalies without burdening storage, making them suitable for continuous monitoring at scale. ### Types of Metrics in OpenTelemetry: * **Counter**: Measures occurrences or events, such as the number of requests handled. * **Gauge**: Captures values that fluctuate, like memory usage. * **Histogram**: Measures the distribution of values, such as response time percentiles. Explore further in the [OpenTelemetry Metrics Documentation](https://opentelemetry.io/docs/concepts/signals/metrics/). ## Why Metrics Matter In a **microservices** environment, metrics are indispensable for: * **Performance monitoring**: Identifying bottlenecks or degraded performance. * **Capacity planning**: Forecasting when additional resources are required. * **Incident detection**: Alerting teams about abnormal system behavior. Metrics are often **the first step** in identifying that something has gone wrong. If a metric shows unusual values (e.g., a spike in response time), you can investigate further by drilling into traces or logs to find the root cause. ## Metrics vs. Traces Metrics have a number of advantages over tracing. Metrics are much more data efficient, generally at the collector level it’s possible to compress hundreds of individual metrics reported to a single packet of data sent on to the metrics backend. Further, metrics show broad trends whereas a trace, no matter how interesting, will always cover only a single request. Should you use metrics instead of traces to monitor your service? Absolutely not. Metrics will always present average performance, and the specific information needed to really understand root causes will be elusive. Further, even with high resolution timeseries metrics it’s very hard to go from worrying metrics to find matching log data of a problem. Finally, modern traces can effectively show information about asynchronous requests as they contribute to overall request time, something that’s very hard to tease out of bare metrics. ## Setting up OpenTelemetry Metrics ### Auto-Instrumentation vs. Manual Instrumentation 1. **Auto-Instrumentation**: Many popular frameworks and libraries come with automatic OpenTelemetry instrumentation, requiring minimal setup. 2. **Manual Instrumentation**: Developers can manually add metrics within the application code by using SDKs to track specific business metrics (e.g., purchases per hour). Learn more about instrumentation options in the [OpenTelemetry SDK Guide](https://opentelemetry.io/docs/instrumentation/). ## Example Metric Pipeline With OpenTelemetry, you can collect, process, and export metrics using **Collectors**. Here’s a high-level example of a typical metric pipeline: 1. **Data Collection**: Metrics are generated by instrumented services. 2. **Processing**: The OpenTelemetry Collector aggregates and processes the data (e.g., batching or filtering metrics). 3. **Exporting**: Metrics are sent to observability platforms like **Prometheus** or **Grafana**. Learn how to configure a collector in the [OpenTelemetry Collector Guide](/docs/learn/opentelemetry/what-is-the-otel-collector/). ## Best Practices for Metrics in OpenTelemetry * **Optimize cardinality**: Avoid creating too many distinct labels, as this can overwhelm storage and query systems. * **Set appropriate aggregation intervals**: Batch data intelligently to balance between real-time insights and system load. * **Use meaningful names**: Clearly describe the purpose of each metric to make dashboards and alerts easier to understand. * **Standardize naming early**: While OpenTelemetry defines standard language for a number of concepts, actual metric naming is not standardized. As such it's possible to report `total-web-shop-checkout-time` and `webShopCheckoutTime_total` as two totally separate metrics even though they should be aggregated. No standard is perfect, of course, and to normalize data before it's stored, use the [filtering tools in the OpenTelemetry collector](/docs/learn/opentelemetry/otel-filtering/). OpenTelemetry metrics provide a robust foundation for observability, helping teams proactively monitor performance and detect issues before they escalate. With the right setup and tooling, you can gain comprehensive insights into your applications, enabling faster resolution times and improved reliability. # OpenTelemetry Agent Management Protocol Source: https://www.checklyhq.com/docs/learn/opentelemetry/otel-agent-management-protocol The OpenTelemetry Open Agent Management Protocol (OpAMP) is an emerging standard aimed at managing telemetry agents, such as the OpenTelemetry Collector, at scale. ## An introduction to OpenTelemetry's new project to allow control of collectors at scale Managing the OpenTelemetry a large number of OpenTelemetry collectors isn’t always easy. In my informal conversations at events like OpenTelemetry community day at KubeCon, most operators are still manually editing the configuration templates for their collectors, and then restarting the process to get it to pick up. If you’re running one collector or even five this isn’t that arduous (in the case of five collectors it’s likely each one has some unique settings), but for hundreds of collector processes on a large cloud this will get tedious fast. For those medium-to-large collector implementations there’s hope: the Agent Management Protocol. The OpenTelemetry Open Agent Management Protocol (OpAMP) is an emerging standard aimed at managing telemetry agents, such as the OpenTelemetry Collector, at scale. OpAMP is designed to allow dynamic configuration, credential management, and remote updates, ensuring observability infrastructure can adapt to changing needs without downtime. While still in development, the protocol holds significant potential for streamlining the management of distributed telemetry agents. ### What is OpAMP? OpAMP provides a standard framework for remotely managing telemetry agents. Although current implementations primarily focus on the OpenTelemetry Collector, the protocol is flexible enough to manage various telemetry-related components, including instrumentation agents or future SDK-level integrations. The primary goal is to allow real-time configuration changes, package updates, and secure connection management without manual intervention. ### Key Features of OpAMP 1. **Dynamic Configuration Management:** OpAMP supports dynamic configuration changes for telemetry agents, such as modifying how data is collected or processed. This is especially useful for managing configurations like sampling rates and trace selection strategies. 2. **Package Management for Auto-Updates:** The protocol allows agents to receive new versions or updates through downloadable packages, including checksums for integrity verification. A supervisor process can handle package downloads, restart the agent with the new version, and roll back to a stable version if issues arise. 3. **Credential Management:** OpAMP facilitates the management of client-side TLS certificates and other credentials. Credentials can be rotated dynamically, either through server-initiated updates or client-initiated requests, ensuring secure communication across agents and servers. 4. **Tail-Based Sampling:** Tail-based sampling, a feature OpAMP aims to support, allows agents to decide which traces to retain after they are fully captured, rather than at the start. This strategy helps reduce the amount of irrelevant telemetry data, though it requires specialized architectures to handle sampling decisions across distributed collectors. ### Deployment Models for OpAMP OpAMP supports multiple deployment models to fit different environments, including: * **Integrated Client Model:** In this model, the OpAMP client runs within the same process as the OpenTelemetry Collector. The client communicates with the OpAMP server to receive updates and reconfigures the collector without restarting the entire process. * **Supervisor Process Model:** The supervisor approach runs the OpAMP client as a separate process alongside the collector. It monitors the collector’s health, applies configuration changes, and ensures that the collector restarts if it crashes or fails to start after updates. This model provides greater fault tolerance, especially in Kubernetes environments where the client and collector can run as sidecars in the same pod. ### Use Cases for OpAMP * **Remote Configuration and Management:** Administrators can adjust configurations across a fleet of agents without restarting them. For example, switching from head-based to tail-based sampling or changing export endpoints can be performed on the fly. * **Resilient Agent Operation:** The supervisor model ensures that agents stay operational, even if configurations cause issues. If a new configuration leads to a crash, the supervisor can revert to the previous stable configuration. * **Credential Rotation for Secure Communication:** TLS certificates and other credentials can be dynamically rotated, ensuring secure and compliant communication between agents and backends. ### Challenges and Future Developments Some advanced use cases, such as tail-based sampling, present challenges due to the need for stateful processing and coordinated decision-making across multiple collectors. While the protocol provides a foundation for such features, implementing them in a distributed environment requires careful architectural design. OpAMP is still evolving, with many features in beta or under active development. In the coming quarters, more user-facing tools are expected, including graphical interfaces for managing collector configurations. As vendors begin to adopt the protocol, end-users can expect smoother agent management and the ability to make configuration changes in real-time. ### The Current State of OpAMP Development The Open Agent Management Protocol (OpAMP) is evolving as a key component for managing fleets of telemetry agents, such as the OpenTelemetry Collector, at scale. As of late 2023, the protocol is still in its **beta phase**, with development focused on enabling dynamic configuration, remote updates, and secure credential management across distributed agents. **Recent Progress**: OpAMP is being implemented in two modes for the OpenTelemetry Collector: 1. **Collector Extension**: A lightweight client-side implementation embedded within the collector to support essential OpAMP operations. 2. **Supervisor Process**: An external process that works alongside the collector to handle more advanced features, such as reverting failed configurations and coordinating updates. This supervisor ensures the collector remains operational by restarting or rolling back if new configurations cause issues. ### **Challenges and Outlook** One key development area is **tail-based sampling**, which requires careful architectural design to coordinate sampling decisions across multiple agents and avoid bottlenecks. The protocol’s potential for dynamic configuration management is also expanding, with planned improvements for user-facing tools, including a graphical interface for managing collector settings. Looking ahead, the next few quarters are expected to bring **more production-ready implementations**. This may include interfaces for real-time configuration changes through OpAMP and better integration within platforms like Kubernetes via sidecars or the OpenTelemetry Operator. ### Conclusion OpAMP represents a significant step forward in managing telemetry agents at scale, offering dynamic configuration, secure credential handling, and automated updates. Although the protocol is still maturing, it promises to simplify the management of observability infrastructure, making it more adaptable and resilient. Organizations implementing OpenTelemetry can look forward to enhanced capabilities for managing their telemetry systems remotely and efficiently. # Advanced Data Filtering with the OpenTelemetry Collector Source: https://www.checklyhq.com/docs/learn/opentelemetry/otel-filtering The OpenTelemetry Collector is a stand-alone service designed to collect, process, and export telemetry data such as logs, metrics, and traces. You can filter this data with the collector. An advanced guide to filtering data with processors and the OpenTelemetry Transform Language. The **OpenTelemetry Collector** is a powerful tool for managing telemetry data flows. One of its most valuable features is the ability to filter and transform data without modifying application code. This flexibility makes the collector an essential component for **controlling data overhead**, ensuring data privacy, and maintaining efficient observability pipelines. *** ## Why Filtering Matters Instrumentation often generates **noisy or redundant data**. Some metrics or traces may include unnecessary information—like personal data or debug logs—leading to bloated dashboards, higher storage costs, and privacy risks. Filtering data within the OpenTelemetry Collector allows you to: * **Remove unwanted data** to reduce noise. * **Mask or drop sensitive information** such as personally identifiable information (PII). * **Transform data formats** (e.g., unify naming conventions). * **Control data flow to different backends** by sending filtered data to specific pipelines. *** ## How the OpenTelemetry Collector Filters Data The Collector consists of **receivers, processors, and exporters**. Each plays a role in data filtering: 1. **Receivers**: Collect data from sources like applications and agents. 2. **Processors**: Modify, filter, or batch data before it is exported. 3. **Exporters**: Send the processed data to observability backends, such as **Prometheus** or **Grafana**. ### Processor-based Filtering Processors allow transformations and data filtering mid-pipeline. Processors offer inherently limited transformation configuration to do the most common tasks like: * **Drop attributes or entire spans**: Remove unnecessary fields or reduce trace volume. * **Normalize data formats**: Convert snake\_case to camelCase, or unify attribute labels. * **Mask sensitive data**: Scrub PII before it is transmitted. Example configuration to normalize naming conventions: ```yaml theme={null} processors: metricstransform/rename: transforms: include: system.cpu.usage action: update new_name: system.cpu.usage_time ``` *** ## Transformations Using OpenTelemetry Transform Language (OTTL) The **OpenTelemetry Transform Language (OTTL)** provides enhanced flexibility for complex transformations. OTTL allows for: * **Pattern-based transformations**: Modify data dynamically using regex or templates. * **Extract and relocate attributes**: Move user IDs from URL parameters to span attributes. * **Conditional filtering**: Drop traces based on specific conditions, such as redundant spans. Here’s an example of the more complex logic available with OTTL, keeping data from app1, app2, OR app3 ```yaml theme={null} processors: filter/ottl: error_mode: ignore traces: span: - | resource.attributes["service.name"] != "app1" and resource.attributes["service.name"] != "app2" and resource.attributes["service.name"] != "app3" ``` Since OTTL allows functional changes of your data, this is really the simplest possible example of a use case: a gate that goes just beyond simple filtering. ## Stateful Operations and Their Limitations While the collector excels at real-time data transformations, it is **stateless by design** to ensure high performance. Stateful operations—such as aggregating historical data—are not natively supported. This ensures the collector remains lightweight and efficient. Common limitations include: * **No support for complex span re-linking**: Spans cannot be re-associated once collected. * **Limited historical context**: Filtering decisions are based only on the current data batch. Note: if you are reading this section trying to figure out how to add stateful filtering, e.g. forwarding only the longest trace seen in a 24-hour period I recommend that you *stop* pursuing this within the OTel collector. The limitations of the collector exist for performance reasons, and adding state to the tool will cause unexpected downstream effects as, for example, the collector becomes too burdened to always be ready to accept OTLP data from your services. ## Filtering Use Cases 1. **PII Masking**: Remove personal data before sending traces to public dashboards. 2. **Debug Data Suppression**: Drop debug-level logs from production pipelines. 3. **Multi-pipeline Export**: Send sanitized data to public dashboards while retaining full data for internal analysis. *** ## Conclusion The OpenTelemetry Collector provides a robust framework for **controlling data flow and transformation** without requiring changes to application code. Its **processor-based filtering** and support for **OTL** make it an ideal tool for managing telemetry data in complex observability setups. By leveraging filtering effectively, teams can reduce overhead, ensure data privacy, and maintain efficient monitoring pipelines. For more details on configuring the OpenTelemetry Collector, explore the [official documentation](https://opentelemetry.io/docs/collector/). # Learn How to Observe and Monitor your Software with OpenTelemetry Source: https://www.checklyhq.com/docs/learn/opentelemetry/overview Learn OpenTelemetry with Checkly. Add monitoring to every piece of your stack with the open standards and open-source tools. Learn Observability and OpenTelemetry with Checkly. Add monitoring to every piece of your stack with the open standards and open-source tools. ## Getting Started Learn the basics of observability and how to get started with OpenTelemetry. Learn how to instrument your application and get deep insights into your stack. ## Diving Deeper Learn about metrics and how to use them to monitor your application. Learn about tracing and its importance in monitoring your application. # A guide to the OpenTelemetry Trace API Source: https://www.checklyhq.com/docs/learn/opentelemetry/trace-api The OpenTelemetry Trace API provides developers with the tools needed to create, manage, and analyze traces, a critical component of observability. ## Taking direct control of OpenTelemetry traces via the API The OpenTelemetry Trace API provides developers with the tools needed to create, manage, and analyze traces, a critical component of observability. Traces represent the journey of a request or process as it flows through a system, breaking it into smaller operations called spans. Understanding how to use the Trace API effectively helps teams gain insight into their systems’ performance and quickly identify bottlenecks or failures. ### Overview The API allows you to: * **Add spans**: Define individual operations within a trace. * **Attach attributes and events**: Provide context to spans, making the trace more informative. * **Manage context propagation**: Maintain relationships between spans across different parts of your code or services. * **Use manual and automatic instrumentation**: Complement automatic tracing with custom spans to capture essential information that auto-instrumentation may miss. ### Manual Instrumentation and Spans While automatic instrumentation can kick off traces and capture key operations, developers often need to add spans manually. For example, a trace may start automatically when handling a web request, but you may want to add spans within specific parts of the application that require deeper insight. Manual spans allow teams to capture contextual information like user IDs or transaction details, providing richer insights for debugging and analysis. Spans are always linked to a **tracer object**, which is provided by a **tracer provider**. This structure allows modular instrumentation, enabling different components of a system to use their own tracers, scoped by version or module, if needed. ### Context Propagation Context propagation ensures that spans within a trace maintain their parent-child relationships, even across different services or functions. The API offers two types of context propagation: * **In-process propagation**: Links spans within the same service or function. * **Out-of-process propagation**: Connects spans across services via trace context headers. In languages like Python or JavaScript, this propagation happens implicitly, but in Go, developers must handle it manually using functions like `SpanFromContext` and `ContextWithSpan`. ### Key Operations on Spans * **SetAttributes**: Attach key-value pairs to spans, providing additional context. * **AddEvents**: Record events within a span’s lifetime to capture specific actions or failures. * **SetStatus**: Indicate whether an operation succeeded or failed, aiding in later troubleshooting. * **AddLinks**: Link different traces together, useful in asynchronous scenarios like message queues. * **RecordException**: Log exceptions as events within spans, including stack traces and error messages. ### Practical Considerations Using the Trace API effectively involves balancing automatic and manual instrumentation. Auto-instrumentation handles common scenarios, but manual spans offer the flexibility to capture unique business logic or operations. Proper use of context propagation ensures that traces remain complete and meaningful, avoiding broken or orphaned spans. The OpenTelemetry Trace API is a powerful tool for teams aiming to build robust observability practices, offering fine-grained control over how operations are tracked and analyzed. By combining this API with backend tools like Prometheus and Grafana, developers can visualize and act on telemetry data to improve system performance and reliability. # What are OpenTelemetry Traces? - key concepts and examples Source: https://www.checklyhq.com/docs/learn/opentelemetry/traces OpenTelemetry traces capture how individual operations within your system interact over time. A trace follows a request as it flows through a system, recording the relationships between different operations. Traces are particularly useful in distributed systems, where multiple services or components interact. However, they are equally valuable for monolithic applications, providing insights even when everything runs in a single process. Learn more about OpenTelemetry & Monitoring with Checkly. Explore metrics, one of the three pillars of observability. OpenTelemetry traces capture how individual operations within your system interact over time. A trace follows a request as it flows through a system, recording the relationships between different operations. Traces are particularly useful in distributed systems, where multiple services or components interact. However, they are equally valuable for monolithic applications, providing insights even when everything runs in a single process. ## Key Concepts in OpenTelemetry Traces 1. **Spans:** * The core unit in a trace. * Represents an individual operation. * Each span has a name, a start and end time, and metadata (attributes) as key-value pairs. * Spans can be nested to reflect parent-child relationships. 2. **Trace Context:** * Propagates trace identifiers across process boundaries. * Helps track related spans across multiple services or components. 3. **Automatic Instrumentation:** * Most languages and frameworks allow tracing without code changes by using instrumentation agents. * This approach quickly provides a basic trace structure, capturing incoming requests and outgoing responses. 4. **Manual Instrumentation:** * Developers use OpenTelemetry APIs to create spans where deeper insights are needed. * Useful for tracking specific application logic or attaching custom attributes. ## Tracing in Monolithic vs. Distributed Systems Though OpenTelemetry is often associated with microservices, its principles apply equally to monoliths. Even when working with a single application, external dependencies like databases, message queues, or third-party services make distributed tracing beneficial. Instrumenting a monolith provides visibility into which operations are slow, how many database calls occur per request, and which API calls contribute to latency. ## Logs and Traces: A Complementary Approach Organizations often have an existing logging infrastructure when adopting tracing. OpenTelemetry’s logs bridge allows integration between structured logs and traces by wrapping logs with trace identifiers. This ensures logs and traces remain correlated without requiring a complete overhaul of existing logging practices. The vision of OpenTelemetry logging remains unfinished, but there's hope in coming years that more tools will exist to connect logs and traces more consistently. Further, until tracing coverage is perfect and well-annotated, logs will be part of the solution to finding the root cause of the problem, after tracing has given strong clues as to the cause (for example with a particularly long and slow span). ### Gradual Migration with Logs Bridge Organizations can slowly convert significant logs into spans, by processing either within their datastore or at logging time. This approach allows gradual adoption of tracing without interrupting existing workflows, ensuring a smooth transition. ## Benefits of OpenTelemetry Tracing * **Visibility:** Quickly identify slow or failing operations. * **Efficiency:** Diagnose complex issues by tracking dependencies and relationships. * **Onboarding:** Help new developers understand system behavior through visualized traces. * **Adaptability:** Works across monoliths, microservices, and hybrid systems. ## Getting Started To begin, select your language and follow the documentation to add [automatic instrumentation](/docs/learn/opentelemetry/how-to-instrument/). All major web development language libraries already support tracing out-of-the-box, making it easier to adopt tracing incrementally. Incorporating OpenTelemetry traces helps developers detect problems earlier, understand their systems better, and respond effectively to user issues. Whether your application is a monolith, a microservice, or somewhere in between, traces provide the insight you need to optimize and troubleshoot your software. # Introduction to the OpenTelemetry Collector Source: https://www.checklyhq.com/docs/learn/opentelemetry/what-is-the-otel-collector The OpenTelemetry Collector is a stand-alone service designed to collect, process, and export telemetry data such as logs, metrics, and traces. It provides a vendor-neutral way to manage this data, offering flexibility in configuration and deployment. Learn more about Playwright & Monitoring with Checkly. Explore how to automate your web with a reliable, programmable monitoring workflow. The OpenTelemetry Collector is a stand-alone service designed to collect, process, and export telemetry data such as logs, metrics, and traces. It provides a vendor-neutral way to manage this data, offering flexibility in configuration and deployment. The collector is extremely lightweight, and can run in almost any environment without significant infrastructure overhead. ## Setting Up Observability with OpenTelemetry ### Instrumentation To begin, you need to instrument your code with OpenTelemetry client libraries. These libraries help generate telemetry signals such as logs, metrics, and traces from your application. ### Data Collection and Processing Once the telemetry data is generated, it can be exported directly to a backend or processed through the OpenTelemetry Collector. Using a collector helps offload the responsibility of data management from the application, making it easier to handle different data pipelines. To use the collector, after setting up your first collector instance, you'll configure your application's OpenTelemetry installation to send data to your collector, generally via the OpenTelemetry protocol or OTLP. ### Deployment Options The OpenTelemetry Collector can be deployed in multiple ways: * **As an agent:** Installed on the same host as the application reporting data. Generally one collector per application the agent model for the Otel collector

* **As a standalone service with a gateway:** Runs independently, receiving telemetry from multiple sources, possibly with a load balancer. the gateway model for the Otel collector

the gateway model for the Otel collector

*There may be a load balancer between all applications and multiple collectors, but this is the simplest version* In larger deployments, a combination of agents and standalone services may be employed to manage scale. ### Data Storage The OpenTelemetry collector is fully stateless, it produces no dashboards and stores no data. It doesn't even create an API endpoint to get status information. The collector is only useful when transmitting data, so your OpenTelemetry data needs somewhere to go. SaaS tools like Coralogix can receive your data, or you'll need to set up your own datastore with something like Prometheus. ## OpenTelemetry and the CNCF The OpenTelemetry framework is part of the Cloud Native Computing Foundation (CNCF) and aims to standardize the handling of telemetry data. It provides a consistent interface to collect and export data across many programming languages. The collector is an important part of this mission, since a standardized proxy for all OpenTelemetry data helps different teams in different languages form a shared understanding of how their data is collected, processed, and transmitted. ### Key Functions of the OpenTelemetry Collector * **Collection:** Collects telemetry data in various formats from multiple sources. * **Processing:** Applies transformations, such as sampling, batching, or removing sensitive data. * **Exporting:** Sends telemetry data to different backend systems. ## Components of the OpenTelemetry Collector 1. **Receivers:** These accept data into the collector. Common formats include OTLP, Jaeger, and Prometheus. 2. **Processors:** Apply operations on the data, such as batching, retries, or adding metadata. Processors can also handle privacy-related tasks like removing personally identifiable information (PII). See our [advanced guide on filtering data with the OpenTelemetry collector](/docs/learn/opentelemetry/otel-filtering). 3. **Exporters:** These send data to backends. You can send metrics to tools like Prometheus and traces to Jaeger or other supported systems. ## Configuring the OpenTelemetry Collector The configuration is managed via a YAML file, defining how receivers, processors, and exporters are connected in pipelines. ### Example Configuration ### Receivers ```yaml theme={null} receivers: otlp: protocols: grpc: http: prometheus: config: scrape_configs: - job_name: 'otel-collector' scrape_interval: 5s static_configs: - targets: ['0.0.0.0:8888'] ``` There are a large number of receivers for multiple formats of data coming in to an OpenTelemetry system. Note that, sadly, not all Prometheus data can be translated 1:1 into OTLP. The Prometheus receiver on Github is [currently listed as a work in progress](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md). ### Processors ```yaml theme={null} processors: batch: send_batch_size: 1000 timeout: 10s memory_limiter: ballast_size_mib: 683 limit_mib: 1500 check_interval: 5s queued_retry: num_workers: 4 retry_on_failure: true ``` Processors are the most diverse component within a collector, with a processor doing anything from removing sensitive data, batching data for transmission, or filtering unwanted information. These examples are all about how data will be batched, the max memory to use (after which data will be thrown out in this reporting cycle), and retries for sending data. ### Exporters ```yaml theme={null} exporters: prometheus: endpoint: "localhost:9090" jaeger: endpoint: "localhost:14250" ``` At their most basic, exporters will contain where the data is headed. ### Pipelines ```yaml theme={null} pipelines: traces: receivers: [jaeger, otlp] processors: [batch, memory_limiter] exporters: [jaeger] metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] ``` Pipelines a path from receiver to exporter, and any data will go through the processors listed in order, left to right. ## Backend Options for OpenTelemetry Metrics and Traces After setting up the OpenTelemetry Collector, you will need tools to visualize and analyze the data. Prometheus is a popular choice for storing and querying metrics, while Grafana can visualize these metrics with dashboards. For tracing data, Jaeger is commonly used. * **Prometheus:** Ideal for collecting and storing time-series metrics. * **Grafana:** A visualization tool compatible with Prometheus metrics. * **Jaeger:** Focuses on distributed tracing for microservices-based applications. ## Use Cases The OpenTelemetry Collector is useful for: * **Redacting sensitive data:** Ensure compliance with privacy and security standards. * **Batching telemetry data:** Compress data and reduce the number of outgoing connections. * **Customizing telemetry pipelines:** Modify attributes, transform metrics, or sample traces to suit your observability needs. ## FAQs ### Do I have to deploy an OpenTelemetry collector? You don't have to use a collector to gather OpenTelemetry data. In the simplest example where you're monitoring a single monolithic application, the difference without a collector would be at the time of reporting data: instead of traces, logs, and metrics going to a collector; data would travel directly to the storage endpoint. If you're starting out with a demo project, or you're only using OpenTelemetry for a very limited purpose (for example you're only implementing [OpenTelemetry to send a few key traces to Checkly](/docs/resolve/traces/how-it-works)), it may make sense not to use a collector at all. ### What is the difference between the OpenTelemetry Agent and Collector? There isn't an 'agent' as such that's part of the OpenTelemetry model. As mentioned above the collector may be deployed in an 'agent' pattern where the collector is running on the same host as the application, but the term 'agent' is a little overloaded in observibility and requires brief disambugation. The other part of observability sometimes called an 'agent' is a process running within an application that receives data on the system. For example, automatic instrumentation of Java applications is made possible by the standard `javaagent` jvm argument. Using an agent to observe your application will depend on your language library's implementation. The collector is not an 'agent' running within an application, it runs outside your application and collects and forwards data. ### How does OpenTelemetry compare with Prometheus? Prometheus is focused on metrics, using a pull model for data collection. OpenTelemetry, on the other hand, is a broader framework that handles logs, metrics, and traces and supports multiple backends. ### What is OpenTelemetry's collector-contrib? The `collector-contrib` repository offers [community-contributed components](https://github.com/open-telemetry/opentelemetry-collector-contrib) that extend the capabilities of the core OpenTelemetry Collector. These additions provide more receivers, processors, and exporters to handle different telemetry scenarios. ## Conclusion OpenTelemetry Collector offers a flexible and scalable way to manage telemetry data. By using open-source tools such as Prometheus and Grafana, organizations can build robust observability systems tailored to their needs. # Playwright Assertions - Types & Best Practices Source: https://www.checklyhq.com/docs/learn/playwright/assertions Learn more about Playwright Assertions. Explore different types, common errors, best practices, and how to get started. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. ### What are Playwright Assertions? Playwright is more than a testing framework. With its generalized tool at simulating user behavior in a browser (or via an API), Playwright is better defined as a web automation framework. You can use Playwright to scrape websites, automate form submissions, or any other time it would be helpful to replace a human web user with a robot. For most users, Playwright is intimately associated with end-to-end tests, and for these we must stop our automated actions and check to see if everything is working right. To make an automation into a test, we have assertions. When a hard assertion fails, the execution of a Playwright test stops, and following steps are not run. ### Types of Playwright Assertions Playwright offers a variety of assertion types to accommodate different testing scenarios, and there are two broad categories for flexible, Web-first assertions and rigid, brittle, non-retrying assertions. Web-first assertions return promises, since they will return when they are successful or time out, where as non-retrying assertions throw their response immediately. ### Auto-retrying, Web-first Assertions Auto-retrying assertions automatically recheck conditions until they pass or a timeout is reached. They both make a test more stable, and can make tests execute more quickly, and in the language of game theory should be 'strictly better' than non-retrying assertions. | **Assertion** | **Description** | | --------------------------------------- | ----------------------------------------------------------- | | `expect(locator).toBeChecked()` | element is checked (e.g., for checkboxes or radio buttons). | | `expect(locator).not.toBeChecked()` | element is not checked. | | `expect(locator).toBeDisabled()` | element is disabled. | | `expect(locator).not.toBeDisabled()` | element is not disabled. | | `expect(locator).toBeEditable()` | element is editable. | | `expect(locator).not.toBeEditable()` | element is not editable. | | `expect(locator).toBeEmpty()` | element has no text content. | | `expect(locator).not.toBeEmpty()` | element has text content. | | `expect(locator).toBeEnabled()` | element is enabled. | | `expect(locator).not.toBeEnabled()` | element is not enabled. | | `expect(locator).toBeHidden()` | element is hidden or does not exist in the DOM. | | `expect(locator).not.toBeHidden()` | element is visible. | | `expect(locator).toBeVisible()` | element is visible. | | `expect(locator).not.toBeVisible()` | element is hidden or does not exist in the DOM. | | `expect(locator).toContainText()` | element contains the expected text. | | `expect(locator).not.toContainText()` | element does not contain the expected text. | | `expect(locator).toHaveAttribute()` | element has the expected attribute value. | | `expect(locator).not.toHaveAttribute()` | element does not have the expected attribute value. | | `expect(locator).toHaveClass()` | element has the expected class. | | `expect(locator).not.toHaveClass()` | element does not have the expected class. | | `expect(locator).toHaveCount()` | locator has the expected number of elements. | | `expect(locator).toHaveCSS()` | element has the expected CSS property value. | | `expect(locator).toHaveId()` | element has the expected ID attribute value. | | `expect(locator).toHaveJSProperty()` | element has the expected JavaScript property value. | | `expect(locator).toHaveText()` | element has the expected text content. | | `expect(locator).not.toHaveText()` | element does not have the expected text content. | | `expect(locator).toHaveValue()` | element has the expected input value. | | `expect(locator).not.toHaveValue()` | element does not have the expected input value. | | `expect(locator).toBeFocused()` | element is focused. | | `expect(locator).not.toBeFocused()` | element is not focused. | ### Non-retrying Assertions Non-retrying assertions evaluate conditions only once. They are used when you expect a condition to be deterministic and want to avoid any retry logic. | **Assertion** | **Description** | | ---------------------------------------- | -------------------------------------------------------------------------------- | | `expect(value).toBe()` | value is strictly equal to the expected value. | | `expect(value).not.toBe()` | value is not strictly equal to the expected value. | | `expect(value).toEqual()` | value is deeply equal to the expected value. | | `expect(value).not.toEqual()` | value is not deeply equal to the expected value. | | `expect(value).toBeTruthy()` | value is truthy. | | `expect(value).toBeFalsy()` | value is falsy. | | `expect(value).toBeGreaterThan()` | value is greater than the expected value. | | `expect(value).toBeGreaterThanOrEqual()` | value is greater than or equal to the expected value. | | `expect(value).toBeLessThan()` | value is less than the expected value. | | `expect(value).toBeLessThanOrEqual()` | value is less than or equal to the expected value. | | `expect(value).toMatch()` | value matches the expected regular expression. | | `expect(value).not.toMatch()` | value does not match the expected regular expression. | | `expect(value).toContain()` | Checks if an array or string contains the expected element or substring. | | `expect(value).not.toContain()` | Checks if an array or string does not contain the expected element or substring. | | `expect(value).toHaveLength()` | Checks if an array or string has the expected length. | | `expect(value).toBeNull()` | value is `null`. | | `expect(value).toBeUndefined()` | value is `undefined`. | | `expect(value).toBeDefined()` | value is defined (not `undefined`). | | `expect(value).toBeNaN()` | value is `NaN`. | This makes the most sense when using Playwright to run single end-to-end tests in a controlled environment. If you're using Playwright to run on a cadence and monitor a production system, for example, with [Checkly](/docs/detect/synthetic-monitoring/browser-checks/playwright-support), you generally want to stick with the auto-retrying assertions to avoid unnecessary alerts. ### Negating Assertions Negating matchers allow you to assert that a condition does not hold. For example, you can use `toBeFalsy` or `not.toContain` to make negative assertions (as will be explained later, if you're checking an element directly, probably the latter of these two options will work better as it implements Playwright's auto-retries). ### Soft Assertions Soft assertions won't fail your test immediately, but rather allow the test to continue running after a failure. Using soft assertions is useful for gathering more complete information during test runs. Soft assertions are a critical component of production monitoring, as when measuring things like load times, a failing response time may not mean that we want to stop the check entirely. Any Playwright test with failing soft assertions will still be listed as failed, but the test will still run to completion (or the next hard assertion that fails). ```typescript login.spec.ts theme={null} // if this assertion fails the following actions and // assertions will still run but the test will be marked as failed await expect.soft(page.getByTestId('cookieBanner')).toBeVisible(); ``` ### Checkly, soft assertions, and degraded check states Checkly can use soft assertions as part of [a `degraded` state for Multistep checks](/docs/detect/synthetic-monitoring/multistep-checks/degraded-states). If a check fails soft assertions and there is a call to `markCheckAsDegraded`, the check will be placed in a 'yellow light' category of degraded rather than failing. Instead of categorizing the check as a total failure the 'degraded' state signals that the check encountered issues but still completed execution. Degraded checks have different notification policies. For example: you might have degraded checks report to Slack rather than sending a notification that wakes people up. This is useful for distinguishing between partial service disruptions and complete outages, offering teams a more precise understanding of performance and reliability. If your Playwright test is running on Checkly, the check will enter a 'degraded' state when some soft assertions fail, or if you call `markCheckAsDegraded` in its execution. ```ts soft-assertions.spec.ts theme={null} import { test, expect } from '@playwright/test' import { getAPIResponseTime, markCheckAsDegraded } from '@checkly/playwright-helpers' const baseUrl = 'https://api.spacexdata.com/v3' test("SpaceX-API Dragon Capsules & Next Launch", async ({ request }) => { await test.step('get all capsules', async () => { const response = await request.get(`${baseUrl}/dragons`) // Hard assertion for a 200 status code, this passes! expect(response).toBeOK() // Soft assertion of a very fast response time expect.soft(getAPIResponseTime(response), 'GET /dragons too slow') .toBeLessThanOrEqual(200) return response.json() }) // Mark as degraded if we got any errors. if (test.info().errors.length) { markCheckAsDegraded('Check degraded due to soft assertion failure.') } }) ``` `degraded` using `@checkly/playwright-helpers` states are currently only available for [Multistep Checks](/docs/detect/synthetic-monitoring/multistep-checks/overview) and [Browser Checks](/docs/detect/synthetic-monitoring/browser-checks/overview). In this case if the response has a `200` status code but the API response time is more than 200 milliseconds, the status of this check will be degraded and the slow API response time will be included as a failed step. If the response came back as a non-200 status (indicating an error) then the check would be 'failed,' whether or not `markCheckAsDegraded` was called. You can [configure alert channels](https://www.checklyhq.com/docs/communicate/alerts/channels#managing-alert-channels) to notify you when a check has degraded. ### Custom Assertions While Playwright has almost every scenario for testing supported in its core methods, it remains quite customizable, even [allowing custom assertions](https://www.youtube.com/watch?v=Fs-nM747TY4). * `expect.configure`: Configures custom options for assertions. * `expect.poll`: Polls a condition until it passes or a timeout is reached, useful for dynamic content. * `expect.toPass`: Runs a block of assertions until all pass, [often used for testing flaky or complex scenarios.](https://www.youtube.com/watch?v=8g7FvoRToGo) ### When to Use Soft vs Hard Assertions Soft assertions are helpful for tests where understanding multiple failures is beneficial, such as end-to-end tests with a wide range of checks. Hard assertions are more suitable for critical conditions where a single failure should immediately terminate the test. | **Scenario** | **Use Soft Assertion** | **Use Hard Assertion** | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **End-to-end testing with multiple checks** | Yes. Useful for gathering comprehensive failure data and understanding multiple points of failure. | No. Would stop execution on the first failure, which is not ideal when you want to diagnose multiple issues. | | **Critical conditions where failure must stop execution immediately** | No. Collecting multiple errors is unnecessary and may lead to misleading results. | Yes. Essential for scenarios where a single failure invalidates the entire test. | | **Monitoring a production system (e.g., API response times)** | Yes. Enables the test to continue running, gathering more data while signaling performance degradation. | Also yes! If components like login buttons, critical notifications, or essential UI elements are missing, we should stop the test and notify the team, rather than waiting for all test steps to be attempted. | | **Testing load times or performance benchmarks** | Yes. Helps measure performance without halting the test, giving more insight into overall behavior. | No. Use hard assertions sparingly here; usually not necessary unless a single metric is critical. | | **Asserting non-deterministic or flaky conditions (for example images loading in X seconds)** | Yes. Can handle variations more gracefully, allowing the test to proceed for better overall coverage. | No. Would stop execution unnecessarily and increase the risk of flaky tests. | | **Pre-check for test prerequisites** | No. If a condition must be true for the rest of the test to proceed, use a hard assertion. | Yes. If a pre-requisite check kicks off further steps, all further steps should stop if the rerequisite fails. | | **Validating UI elements during navigation** | Yes. When testing multiple UI states, soft assertions can log failures without interrupting the flow. | Only if the missing UI element would block further navigation. | | **Critical database operation validations** | No. If a data inconsistency occurs, it's usually vital to stop further operations. | Yes. Data integrity should be enforced strictly with hard assertions. | | **Form field validations** | Yes. Allows collecting errors on multiple fields at once, providing more detailed feedback. | No. Stopping at the first validation failure can hinder broader test coverage. | | **Verifying the presence of essential elements before action** | No. If the element must exist for the test to continue, a hard assertion ensures reliability. | Yes, if the element's presence is not critical and you want the test to proceed. | ### How to Get Started with Playwright Assertions in Simple Steps Follow these steps to get up and running with Playwright Assertions: 1. Install Playwright:`npm init playwright@latest`. 2. Start writing tests with various assertion types to validate application behavior. ### How to Add Custom Matchers Using `expect.extend` Custom matchers can be added using `expect.extend`, allowing developers to define more readable or application-specific assertions. ### Playwright Custom Matchers with Example Here's a quick example of how to create a custom matcher for checking specific conditions: ```ts custom-matchers.spec.ts theme={null} expect.extend({ toBeWithinRange(received, floor, ceiling) { const pass = received >= floor && received <= ceiling; if (pass) { return { message: () => `expected ${received} not to be within range ${floor} - ${ceiling}`, pass: true, }; } else { return { message: () => `expected ${received} to be within range ${floor} - ${ceiling}`, pass: false, }; } }, }); // Usage expect(100).toBeWithinRange(90, 110) ``` This can be especially useful if we're doing complex assertions, or specialized parsing of responses. ## Common Assertion Errors and How to Debug Them Understanding how to diagnose assertion failures can save significant debugging time. ### Hard waits *Hard waits should be avoided.* A 'hard wait' refers to giving an exact period of time before going on to the next step of a test. Generally using code like: ```ts bad-practice.spec.ts theme={null} await page.waitForTimeout(3000); ``` When you have page components you want to ensure are loading quickly, or you have page components that you know take some time to load, it's tempting to reach for a hard wait, however, there are better ways to accomplish this functionality, like the soft assertions and auto-retry assertions listed above. After that fixed period, if the next step doesn't work, the test will fail. Hard waits are the most common cause of unreliable or 'flaky' tests with Playwright. They're also inefficient, if the element we're checking for becomes available *before* the hard wait time has elapsed, the test runner still has to wait to the end of the wait time. Read more about this in our page on [waiting in Playwright](https://www.checklyhq.com/learn/playwright/waits-and-timeouts/). ### Manual Assertions While a hard wait is problematic since it doesn't benefit from Playwright's auto-waiting, an improperly structured assertion doesn't include waiting at all. By wrapping the `await` inside the `expect()` instead of vice versa, and by using a non-web-first assertion, we get no waiting. In this case we're checking that there is some text inside an element. ```ts manual-assertion-bad.spec.ts theme={null} // expects the text to be visible instantly expect(await targetPage.locatorToSomeElement.innerText()).toBeTruthy(); ``` Use web-first assertions such as `toBeEmpty()` instead. ```ts web-first-assertion-good.spec.ts theme={null} // uses auto-waiting, defaulting to five seconds max await expect(targetPage.locatorToSomeElement).not.toBeEmpty(); ``` ### Debugging Assertion Errors Playwright provides debug tools such as `page.pause()` and verbose logging to inspect what happens when a test fails. You might also consider a visual like the Playwright VS Code extension to add a visual debugger when writing or debugging tests. [Read more about debugging Playwright scripts](https://www.checklyhq.com/learn/playwright/debugging/). ### Identifying Flaky Tests Flaky tests are those that fail intermittently, often due to timing issues or unpredictable environment factors. You can use `expect.poll` and `.toPass` adjust timeouts to [stabilize flaky tests](https://www.youtube.com/watch?v=8g7FvoRToGo). To identify flaky tests you can use `--repeat-each=100` to really hammer a test and see if it passes constantly. ### Best Practices for Using Playwright Assertions Implementing best practices can make your tests more reliable and easier to maintain. * **Test Granularity**: Structure tests to cover distinct, isolated behaviors, making debugging easier. The choice between one 'super test' and 5 or 10 smaller tests should always go with the more granular option. Since [tests can be run in parallel](https://www.checklyhq.com/learn/playwright/testing-in-parallel/), it's likely that better test granularity will also improve test execution time. * **Create Concise Statements**: Keep assertions simple and focused. Asserting that five different page components is an indicator that you're testing multiple ideas with one test, and want to consider either breaking up your test or should at least have multiple steps. * **Use Descriptive Messages**: Adding clear messages to assertions helps identify failures quickly. It's also good to use [test steps](https://www.checklyhq.com/blog/improve-your-playwright-documentation-with-steps/) to add more description to the exact phase where the test failed. Read further to see how to [add test steps programmatically with Typescript decorators](https://www.checklyhq.com/blog/playwright-test-steps-with-typescript-decorators/). * **Parameterize Assertions**: Use variables for assertion parameters to improve test readability and maintainability. * **Leverage Libraries**: Utilize existing libraries for common assertions where possible. Note that if you're running tests on a cadence, a full set of libraries may not be available in your execution environment. * **Be Specific**: Make sure your assertions are as targeted as possible to avoid false positives. Bet on auto-waiting and web-first assertions. Even a change from checking if a text label has the needed text with [`.toContain()`](https://playwright.dev/docs/api/class-genericassertions#generic-assertions-to-contain-1) rather than `toBe()` can improve the reliability of a page monitor. * **Leverage Built-in Timeout**: Use Playwright's built-in timeout features to handle flaky conditions gracefully. Avoid hard waits whenever possible. ### How Checkly can help with Playwright Checkly enables you to take your existing Playwright code base to run it as global synthetic monitoring. Use the power of Playwright to effectively simulate a real user's behavior with your site or service, and get alerts of problems before your users have noticed.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

### Conclusion By using the right types of assertions, debugging tools, and best practices, you can ensure your tests are robust and efficient. In general, my direct experience has served to emphasize the wisdom of the Playwright project's recommendations: web first assertions, dependent on web-first selectors. For assertions, web-first, auto-retrying assertions are the preferred choice for most scenarios, as they leverage Playwright's built-in waiting mechanisms, making tests more resilient and less prone to flakiness. These assertions continuously check conditions until they pass or reach a timeout, which can help streamline test execution. Examples like `toBeVisible` or `toContainText` illustrate how these assertions can improve test stability. On the other hand, non-retrying assertions evaluate conditions just once and should be reserved for scenarios where retries are unnecessary or could introduce ambiguity. Anything other than web-first assertions will lead to brittle tests with more false positives. Soft assertions also add value by collecting all failures within a block before stopping the test, offering a complete picture of what went wrong. They are especially useful in production monitoring scenarios where partial failures should not disrupt the entire check, as seen in the Checkly example using `markCheckAsDegraded`. The ability to use soft assertions alongside the concept of a degraded state provides a nuanced way to monitor service health without causing false alarms. I'll again recommend avoiding some common pitfalls, such as relying on hard waits, which can introduce flakiness and inefficiency into your tests. Using auto-waiting mechanisms and structuring assertions properly can make a significant difference. Debugging tools like `page.pause()` and verbose logging, combined with visual debugging options, can be valuable in diagnosing test failures. Follow best practices now for a better testing and monitoring experience later. These include using concise, focused assertions, adding descriptive messages for better debugging, and leveraging built-in timeouts and parameterization for clarity and maintainability. Granular test design also matters as a way to improve both parallel execution efficiency and debugging ease. Avoiding complex, monolithic tests in favor of smaller, well-defined ones can make your testing framework more manageable and robust. # How to Manage Authentication in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/authentication Learn how to handle authentication in Playwright, including SSO, 2FA, Passkeys and more. Reuse authentication state across tests with storageState. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. When testing or monitoring web applications like e-commerce sites, SaaS applications or anything that lives behind a login, you will have to deal with authentication. Playwright takes care of a lot of the heavy lifting here, meaning you don’t have to manually get, set and persist cookies or manage local storage data. In this article we will look at some typical authentication scenarios, explain how using Playwright’s `storageState` works under the hood and give some best practices when dealing with authentication data. # Handling authentication flows For web and applications and APIs, you will typically have to run through some interactive authentication flow. Here are the top, common flows. > Use Playwright’s built-in [code generator](https://playwright.dev/docs/codegen-intro#running-codegen) to quickly get a script of your login steps. It will do the right thing in 90% of the cases. Just run `npx playwright codegen ` ## Basic Auth / Username & Password Any flow that relies on the Basic Authentication standard or typing in a username and password will tend to look very similar to example test below. ```ts basic-auth.spec.ts theme={null} import { test, expect } from '@playwright/test' test('login', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() await page.getByPlaceholder('Email').fill(process.env.USER_EMAIL as string) await page.getByPlaceholder('Password').fill(process.env.USER_PASSWORD as string) await page.getByRole('button', { name: 'Sign In' }).click() await expect(page.locator('#login-message')) .toContainText(`Welcome back, ${process.env.USER_EMAIL as string}`) }) ``` Notice the following: 1. We are using the `getBy` locators where we can. 2. The username and password are referenced as environment variables. Never store them in your `.spec.ts` file! 3. At the end, we assert that the login actually worked, by checking for a user specific piece of content on the page. 4. All handling of cookies, hashing of username and password and other things your browser typically takes care of are handled here transparently. ## SSO & Social Login Any SSO or social login — think Google, GitHub, Microsoft, SAML-based solutions or any 3rd party auth provider like Okta— works pretty much the same as basic authentication / username & password from the perspective of the test you are writing. You will probably see some extra redirects as the login attempt is executed on a second domain, but in 99% of the cases Playwright is smart enough to handle these redirects. Awesome. Here is an example of using Google login on stackoverflow\.com. 👇 ```ts google-login.spec.ts theme={null} import { test } from '@playwright/test' test('Google login', async ({ page }) => { await page.goto('https://stackoverflow.com/users/login') await page.getByRole('button', { name: 'Log in with Google' }).click() await page.getByLabel('Email or phone').fill(process.env.GOOGLE_USER as string) await page.getByRole('button', { name: 'Next' }).click() await page.getByLabel('Enter your password').fill(process.env.GOOGLE_PWD as string) await page.getByRole('button', { name: 'Next' }).click() await page.getByRole('button', { name: 'Continue' }).click() }) ``` [Check the full article on Google Login.](https://www.checklyhq.com/learn/playwright/google-login-automation/) ## Two-Factor Authentication / TOTP Things get a little harder when using Two-Factor Authentication (2FA) and / or Time Based One Time Passwords (TOTP). You will probably be familiar with providing an extra “factor” like a code from an SMS message, a authenticator app or email to a login flow. How do we access a text message (or any of the other options) in Playwright? In short, we don’t. To solve this we need to use the excellent [otpauth NPM package](https://www.npmjs.com/package/otpauth). We wrote a full article on [using the `otpauth` package to login to a GitHub account protected by 2FA](https://www.checklyhq.com/learn/playwright/bypass-totp/), so please check out that article for all the details. Below is the eventual `.spec.ts` file you will end up with. ```ts 2fa.spec.ts theme={null} import { expect, test } from '@playwright/test' import * as OTPAuth from 'otpauth' const totp = new OTPAuth.TOTP({ issuer: 'Raccoon', label: 'GitHub', algorithm: 'SHA1', digits: 6, period: 30, secret: process.env.GITHUB_OTP, }) test('GitHub 2FA login', async ({ page }) => { await page.goto('https://github.com/') await page.getByRole('link', { name: 'Sign in' }).click() await page.getByLabel('Username or email address').click() await page .getByLabel('Username or email address') .fill(process.env.GITHUB_USER) await page.getByLabel('Username or email address').press('Tab') await page.getByLabel('Password').fill(process.env.GITHUB_PW) await page.getByRole('button', { name: 'Sign in' }).click() await page.getByPlaceholder('XXXXXX').click() await page.getByPlaceholder('XXXXXX').fill(totp.generate()) await expect(page).toHaveURL("https://github.com") }) ``` Notice the following in this code example: 1. We call the `OTPAuth.TOTP()` function and pass in a secret token we previously got from the GitHub UI. 2. In our test, we run through a fairly normal login routine. In the end, we focus on the field filled with the `XXXXXX` placeholder and call `totp.generate()` to create the 2FA token. As mentioned, there is some prep you need to do. Refer to our full [article on 2FA login](https://www.checklyhq.com/learn/playwright/bypass-totp/) for all instructions. ## Passkey & **WebAuthn** Passkeys are a fairly recent authentication scheme. Passkeys are passwordless, so passwords can’t be stolen or phished. There is also no need to memorize a passkey. Often, a passkey is tied to biometric data: every time you authenticate on some app or service with Face ID or Touch ID or using [YubiKey](https://www.yubico.com/) you are using a passkey. You can see how this is problem when running Playwright. However, under the hood all passkeys are an implementation based on the [Web Authentication API (WebAuthn)](https://webauthn.guide/#about-webauthn) spec and all Chrome Devtools Protocol (CDP) based browsers like Chrome and Edge ship with a WebAuthn Virtual Authenticator. This virtual authenticator allows you to automate the user interactions normally done by an actual human being. This [excellent write up from the folks at Corbado](https://www.corbado.com/blog/passkeys-e2e-playwright-testing-webauthn-virtual-authenticator#webauthn-virtual-authenticator-e2e-passkey-testing) goes into a lot more detail. Let’s look at an E2E example. We will use the [https://webauthn.io/](https://webauthn.io/) site as our testing target and perform the following actions. 1. Enabled WebAuthn. 2. Sign up with a username + passkey. 3. Login with that passkey. ```ts webauthn.spec.ts theme={null} import { test, expect } from '@playwright/test' test('test webauthn', async ({ page }) => { const client = await page.context().newCDPSession(page) await client.send('WebAuthn.enable') const authenticator = await client.send('WebAuthn.addVirtualAuthenticator', { options: { protocol: 'ctap2', transport: 'internal', hasResidentKey: true, hasUserVerification: true, isUserVerified: true, automaticPresenceSimulation: true, }, }) await page.goto('https://webauthn.io/') await page.getByPlaceholder('example_username').fill('felixacademia') await page.getByRole('button', { name: 'Register' }).click() await page.waitForSelector('text=Success! Now try to authenticate...') await page.getByPlaceholder('example_username').fill('felixacademia') await page.getByRole('button', { name: 'Authenticate' }).click() await expect(page.getByRole('heading', { name: 'You\'re logged in!' })) .toBeVisible({ timeout: 15000 }) }) ``` Notice how we first have to set up the WebAuthn virtual authenticator via the CDP session. This allows us to simulate the passkey authentication flow that would normally require user interaction. The code is quite straightforward after that - we just fill in a username and click through the registration and authentication steps. ## API tokens When working with APIs, you most often need to authenticate using some Bearer token / API key. There is no magic here, as Playwright makes it easy to include these tokens in your request headers. ```ts api-token.spec.ts theme={null} import { test, expect } from '@playwright/test' test('API token auth', async ({ request }) => { const response = await request.get('https://jsonplaceholder.typicode.com/posts', { headers: { 'Authorization': `Bearer ${process.env.API_TOKEN}` } }) expect(response).toBeOK() }) ``` Just like with username and password authentication, make sure to store your API tokens securely using environment variables. # Reusing authentication state with `storageState` All of the above examples work fine if you are running just one test that requires authentication. However, the moment you will run more tests — either in parallel or in sequence — you will exercise your authentication endpoint / provider over and over again. This will cause issues with rate limiting and just make your tests run unnecessarily long. Ideally, you can authenticate once and reuse the authenticated state across multiple tests. [Playwright has this feature baked in](https://playwright.dev/docs/auth) and leverages something called `storageState`. It requires a little setup 👇 ## Setting up the `playwright.config.ts` file The recommended way to reuse auth state is by setting up **projects** in your `playwright.config.ts` file and defining a `setup` step that references file — `auth.setup.ts` for example — that takes care of the necessary authentication flow. ```ts playwright.config.ts theme={null} import { defineConfig, devices } from '@playwright/test' export default defineConfig({ projects: [ { name: 'setup', testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, ], }) ``` ## Writing a `auth.setup.ts` file The code in `auth.setup.ts` stores the eventual authenticated state (cookies and local storage) on disk in a user defined file. Any other projects declare an explicit dependency on the `setup` step and read the auth state from disk, defined by the `storageState` property. Playwright then makes sure the cookies and local storage items are handled correctly in subsequent tests. The example below shows how to use the basic authentication login flow from above in the `auth.setup.ts` file. ```ts auth.setup.ts theme={null} import { test as setup, expect } from '@playwright/test' import path from 'path' const authFile = path.join(__dirname, '../playwright/.auth/user.json') setup('authenticate', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() await page.getByPlaceholder('Email').fill(process.env.USER_EMAIL as string) await page.getByPlaceholder('Password').fill(process.env.USER_PASSWORD as string) await page.getByRole('button', { name: 'Sign In' }).click() await expect(page.locator('#login-message')) .toContainText(`Welcome back, ${process.env.USER_EMAIL as string}`) await page.context().storageState({ path: authFile }) }); ``` Notice that: * We just run a normal `test` , renamed to `setup` . * We store the all cookies and local storage items in the `authFile` by calling `.storageState()` . How this happens doesn’t interest us. Playwright takes care of it. ## Reusing authentication in tests Now, on each invocation of `npx playwright test` , the `auth.setup.ts` routine gets called first. This means we can just create a new test as normal and magically the normal `page` fixture is authenticated. ```tsx theme={null} import { test } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('https://danube-web.shop/') // already authenticated }) ``` Have a look at the [official Playwright documentation for more advanced scenarios.](https://playwright.dev/docs/auth#advanced-scenarios) # Best practices for authentication When writing automation scripts that deal with authentication, there are some general principles you should stick to: 1. Use the Playwright Code Generator, `npx playwright codegen example.com` to get the boilerplate code for any authentication flows. Saves time and let’s you focus on the difficult parts. 2. Never, ever store credentials in your tests. Not even during writing and debugging. Always use environment variables like `process.env.MY_PASSWORD` . Use them right at the beginning as you will forget about them and then accidentally `git push` them. 3. Try to always use dedicated test users, not your own account or — god forbid — a customer’s account. This way you can control the test data more easily, not accidentally trigger a lock out due to bot detection. 4. Always add any `.env` files and the `/playwright/.auth/user.json` to your `.gitignore` file. # Further reading * [How to automate Google login with Playwright.](https://www.checklyhq.com/learn/playwright/google-login-automation/) * [How to bypass Time-Based 2FA login flows with Playwright.](https://www.checklyhq.com/learn/playwright/bypass-totp/)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Bypass Time-Based 2FA Login Flows With Playwright Source: https://www.checklyhq.com/docs/learn/playwright/bypass-totp Automate logins protected by time-based one-time password (TOTP) two-factor authentication in Playwright by generating valid passcodes inside your test. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Testing applications behind a login flow is cumbersome. And it gets even worse when there’s two-factor authentication (2FA) involved. Many people work around this problem by disabling it or implementing wild hacks. Automating a 2FA-based login flow is just too hard! I thought that for a long time, too. But I must admit — I was wrong. This tutorial explains how to write a [Playwright](https://playwright.dev/) automation script to log into github.com with an enabled time-based one-time password (TOTP) two-factor authentication. But before getting into code, let’s look at how this 2FA method works! ## The goal: Make Playwright log into a 2FA-secured github.com account Whether you’re building a fancy SaaS product or an e-commerce solution, security must be a major concern because you want to ensure (maybe even enforce) that your customers' accounts are safe. Two-factor authentication is an additional way to secure all your user accounts. In a nutshell, 2FA pairs a username/password combination with another factor. This other factor could be something you know, have, or are. And while you could go the fancy route of looking into implementing iris scans or fingerprints (“something your users are”), I doubt people will appreciate your efforts. That’s why the three most common 2FA methods are SMS, native apps, or authenticator apps with software tokens. Following the flow, people log in with their credentials and are prompted with another confirmation dialog — the second factor. After receiving and entering the second-factor passcode, they can confirm that they really are the person they claim to be and access their account. And indeed, testing a passcode coming from a mobile application or SMS is challenging, but as it turns out, authenticator apps don’t include as much magic as I initially thought! ### Authenticator apps and TOTP Let’s look at GitHub’s 2FA flow. When setting up 2FA with an authenticator app, the service greets you with the following screen. an authentication QR code

*Note that I blacked out the QR code not to leak any credentials.* The displayed QR code encodes the following information: otpauth://totp/GitHub:USERNAME?secret=SECRET\&issuer=GitHub This encoded URL includes: * a protocol (`otpauth`) to signal that this URL should be opened with an authenticator app * the type (`totp`) * a label (`GitHub:USERNAME`) that is a colon-separated combination of issuer and username * a secret (`SECRET`) * the issuer (`GitHub`) **The essential piece to logging in and automating the TOTP flow is the provided secret.** You can access this secret in the Github UI by clicking 'enter this text code' to view the plain text version of the secret. a two-factor secret

But how will this secret be used? The secret is paired with the current time to generate a one-time password in a TOTP flow. If you scan the QR code that encodes the URL (`otpauth://totp/…`) with an authenticator app: the app stores the second-factor secret, displays the issuer and username, and generates a new passcode every 30 seconds (30 seconds is the default but can be changed). That’s all! You then can use the passcode as a second login factor and GitHub, as our example, can be sure that it’s you accessing your account. To be honest, I was a little underwhelmed when I learned how this works, but it also means that this flow can easily be automated. Let’s get into some browser automation with Playwright! ## Testing a TOTP login flow with Playwright Now that you understand that TOTP flows are based on the current moment and a secret stored on a device, you can start automating it. ### Prerequisites The following steps assume you have created a GitHub account, know its username and password combination, and have access to the account’s TOTP secret. If you have all this, let’s go! ### Setting the foundation with code generation The easiest way to [create a new Playwright script is by using its code generation feature](https://playwright.dev/docs/codegen). `npx playwright codegen github.com` When you run this command, it will open a new browser session, and all your interactions will be recorded in the Playwright Inspector.

Using codegen, you can record the first steps in a breath. ```js theme={null} // tests/2fa.spec.js import { test, expect } from '@playwright/test' test('test', async ({ page }) => { await page.goto('https://github.com/') await page.getByRole('link', { name: 'Sign in' }).click() await page.getByLabel('Username or email address').click() await page.getByLabel('Username or email address').fill('username') await page.getByLabel('Username or email address').press('Tab') await page.getByLabel('Password').fill('password') await page.getByRole('button', { name: 'Sign in' }).click() await page.getByPlaceholder('XXXXXX').click() }) ``` The script above navigates to github.com, clicks the login button and enters the login credentials. But because the account is secured via 2FA, we’re now prompted to enter our one-time passcode. Note [that thanks to Playwright’s auto-waiting feature](https://www.youtube.com/watch?v=j-QLpb6Tmg0\&list=PLMZDRUOi3a8NtMq3PUS5iJc2pee38rurc\&index=3), there is no need to wait for navigations or URL changes. Playwright handles all the waiting for you — magic! playwright codegen

But how can you create the authentication code in your Playwright script? ### Using a TOTP library to generate the passcode Luckily, the thriving npm ecosystem has a solution for every problem. The [OTP Auth library](https://www.npmjs.com/package/otpauth) has 40k weekly downloads and is in its 9th major version, which makes it a good choice that provides all the functionality to generate one-time passwords. Install it via npm: `npm install otpauth` Then, initialize a new totp instance in your Playwright script… ```js theme={null} // generate a new totp instance const totp = new OTPAuth.TOTP({ issuer: "GitHub", label: "USERNAME", algorithm: "SHA1", digits: 6, period: 30, secret: "XXXXXXXX", }) ``` And fill Github’s passcode input field with a newly generated token! ```js theme={null} // generate the passcode when filling the input field await page.getByPlaceholder("XXXXXX").fill(totp.generate()) ``` Note that GitHub automatically detects input field changes, so there’s no need to click the verify button. And that’s it! Now you only have to clean up the hardcoded tokens and replace them with environment variables, and you just created a Playwright script that logs into GitHub and passes the 2FA check! Here’s the final script. ```js theme={null} import { expect, test } from "@playwright/test" import * as OTPAuth from "otpauth" const totp = new OTPAuth.TOTP({ issuer: "Raccoon", label: "GitHub", algorithm: "SHA1", digits: 6, period: 30, secret: process.env.GITHUB_OTP, }) test("GitHub 2FA works", async ({ page }) => { await page.goto("https://github.com/") await page.getByRole("link", { name: "Sign in" }).click() await page.getByLabel("Username or email address").click() await page .getByLabel("Username or email address") .fill(process.env.GITHUB_USER) await page.getByLabel("Username or email address").press("Tab") await page.getByLabel("Password").fill(process.env.GITHUB_PW) await page.getByRole("button", { name: "Sign in" }).click() await page.getByPlaceholder("XXXXXX").click() await page.getByPlaceholder("XXXXXX").fill(totp.generate()) await expect(page).toHaveURL("https://github.com") await page.screenshot({ path: "home.png" }) }) ``` ## Conclusion I’m still amazed at how far we’ve come with the recent tooling. Playwright continues pushing out monthly releases, and I can’t wait to see what they’ll come up with in the upcoming months. But keep in mind end-to-end testing is only a tiny fraction of guaranteeing that your product is working. Third-party providers can go rogue, or your database can struggle days after you tested your deployment. That’s why I’m excited to announce that [the recently released Checkly runtime (2022.10)](/docs/platform/runtimes/runtime-specification#2022.10) includes the “otpauth” package and allows you to test your products at all times — even the ones that are behind a 2FA secured login! Do you want to be the first one to know when something’s off with your application? Give Checkly a try 😉.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Handle Challenging Automation Flows in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/challenging-flows Automate hard-to-test flows like signup, social login, and password reset in Playwright, and get past captchas and bot detection in test environments. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. While automation tools are fundamental to modern software development, they also have the innate potential to be used for malicious purposes. As a consequence, some user flows are made purposely hard to automate to defend against threat actors. Some examples: 1. New user signup 2. Social login through providers like Google, Facebook etc. 3. "Forgot password" and similar reset scenarios There are several means through which automation is made more difficult. ## Captchas Captchas are a popular measure many websites take in order to counter automation for sensitive user flows, with Google's ubiquitous [reCAPTCHA](https://www.google.com/recaptcha/intro/v3.html) being perhaps the most known example. Captchas work by challenging users with tasks that are trivial for humans and hard for bots, like finding objects belonging to the same category in an image or reading obfuscated strings of text. It is beyond the point of a test to try and solve a captcha: we are not trying to verify that the captcha itself works, rather to get past the captcha so we can test the functionality of our target platform. Also, if we were able to automatically solve a captcha, that would defeat the whole purpose of including one in our application. For pre-production environments, we can: 1. [Set captcha to always pass](https://developers.google.com/recaptcha/docs/faq#id-like-to-run-automated-tests-with-recaptcha.-what-should-i-do) 2. Have captchas removed ## Bot detection Certain platforms might have built-in mechanisms to tell manual and automated traffic apart, with the goal of shutting down the latter. These could be based on traffic volume analysis, HTTP fingerprinting, IP blacklisting or even on user behaviour analysis (e.g. the way the user moves the mouse cursor). Headless browsers will normally identify themselves as such in their User-Agent request headers, making detection trivial: ``` Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/79.0.3945.0 Safari 537.36 ``` Bot detection could be deactivated for pre-production environments in order to allow automation. For production environments, a secret may be included in the User-Agent (or similar mechanism) for the system to recognize test bots and allow them through: ```ts set-useragent.spec.ts theme={null} import { test } from '@playwright/test' test.use({ userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/79.0.3945.0 Safari 537.36 Secret/' }) test('my test', async ({ page }) => { await page.goto('https://example.com') // test code }) ``` ## Automation-resistant UIs Certain industries are particularly affected by malicious automation attempts, especially where high-volume automation offers an unfair advantage. Examples of this include online betting and gaming. In these cases, UI automation is specifically made more difficult by utilising components that are harder to interact with, and in some cases by randomising attribute values for key elements in order to prevent referencing. Even if this measure does not defeat all attempts, it can make it excruciatingly hard to produce reliable automated scripts, which might be enough to discourage a large number of attackers. Whenever looking at production environments, keep in mind that adding exceptions to any of the methods listed above might open new attack vectors for malicious users. Always conduct proper risk analysis before proceeding. ## Further reading 1. [OWASP's Automated Threat list](https://wiki.owasp.org/index.php/Category:Automated_Threat)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Mastering E2E Checkout Testing Using Playwright Source: https://www.checklyhq.com/docs/learn/playwright/checkout-testing-guide Write end-to-end checkout tests with Playwright, adding products to a cart, entering billing details, and confirming the purchase completes successfully. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Another key website flow that needs to be closely monitored is any sort of checkout. You want to be 100% sure your users are able to pay for goods or services you might be offering, at any given time. ## Steps Checkout procedures can vary dramatically depending on what is being bought or sold. A few constants will be: 1. a shopping basket page / section, in cases where multiple items can be bought - this is normally where the checkout procedure starts 2. a page to enter or edit billing and, where applicable, shipping information 3. a summary to review all the different parameters of the purchase Modelled on the above structure is the following example running against our test website. We will add a few products to the shopping cart, then proceed until the summary screen shows up and verify that the transaction has been confirmed. Here we can get creative and, for example, iterate through a number of products to fill the cart: ```ts checkout.spec.ts theme={null} import { test } from '@playwright/test' const productsNumber: number = parseInt(process.env.PRODUCTS_NUMBER) || 3 test('checkout', async ({ page }) => { await page.goto('https://danube-web.shop/') for (let i = 1; i <= productsNumber; i++) { await page.click(`.preview:nth-child(${i}) > .preview-author`) await page.getByRole('button', { name: 'Add to cart' }).click() await page.click('#logo') } await page.locator('#cart').click() await page.getByRole('button', { name: 'Checkout' }).click() await page.getByPlaceholder('Name', { exact: true }).fill('Max') await page.getByPlaceholder('Surname', { exact: true }).fill('Mustermann') await page.getByPlaceholder('Address').fill('Charlottenstr. 57') await page.getByPlaceholder('Zipcode').fill('10117') await page.getByPlaceholder('City').fill('Berlin') await page.getByPlaceholder('Company (optional)').fill('Firma GmbH') await page.getByLabel('as soon as possible').check() await page.getByRole('button', { name: 'Buy' }).click() }) ``` Run this example as follows: ```sh theme={null} PRODUCTS_NUMBER=3 npx playwright test checkout.spec.ts ``` ```sh theme={null} SET PRODUCTS_NUMBER=3 npx playwright test checkout.spec.ts ``` Note: In some cases, users will need to [log in](/docs/learn/playwright/login-automation/) before they can proceed to a purchase. When users are allowed to buy both with and without having a pre-existing account on the platform, it might be worthwhile to test both flows separately. ## Limitations Checkout is a peculiar flow: unlike Login and others, it almost always involves an actual exchange of currency. This is usually not an issue in pre-production environments, as payment providers are not yet involved or are set up in sandbox/test mode. For Production, low-value transactions can be performed with low frequency and immediately voided after the test has been completed. ## Takeaways 1. Checkout flows will vary depending on what is being purchased. 2. Remember to check different flows (e.g. with and without login) if needed. 3. Additional care is needed on production systems as real transactions will take place.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Click, Type, and Hover with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/clicking-typing-hovering Replicate real user interactions in Playwright, clicking buttons, typing into fields, and hovering over elements using user-first selectors. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Users normally access most website functionality through clicks, keystrokes etc. Playwright allows us to replicate these events by referencing elements on the page using [User-first Selectors](https://www.checklyhq.com/blog/playwright-user-first-selectors/). ## Clicking Clicking is the default way of selecting and activating elements on web pages, and will appear very often in most headless scripts. ```ts basic-click.spec.ts theme={null} import { test, expect } from '@playwright/test' test('can click log in', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() }) ``` ## Hovering A popular pattern among web pages is exposing additional information or functionality when the user hovers the mouse cursor over a specific item. Examples include, menus, previews and dialogs containing extra information on the item. ```ts basic-hover.spec.ts theme={null} import { test, expect } from '@playwright/test' test('hover over sign in', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() await page.getByRole('button', { name: 'Sign In' }).hover() }); ``` *Note that in this example we're not asserting anything, since our web example doesn't do any element updates on hover.* ## Focussing Focussing on specific UI elements allows the user to interact with them without clicks. It can also result in a proactive reaction from the webapp, such as displaying suggestions. ```ts basic-focus.spec.ts theme={null} import { test, expect } from '@playwright/test' test('Focus on email field', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() await page.getByPlaceholder('Email').focus() }); ``` ## Typing We can simulate typing on a real keyboard using `page.type()`: ```ts basic-type.spec.ts theme={null} import { test, expect } from '@playwright/test' test('testAlpha', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('textbox').focus() await page.getByRole('textbox').fill('catcher') await page.getByRole('button', { name: 'Search' }).click() await expect(page.locator('#app-content')).toContainText('Rye') }); ``` Single key presses can also be executed. For example, to press the Enter key: * Playwright: `await page.press('Enter')` Key presses can also be sent to a specific element: `await (await page.$('input[type="text"]')).press('Enter')` We can also hold down and release one or more keys, possibly combining them to use keyboard shortcuts: ```ts theme={null} await page.keyboard.down('Control') await page.keyboard.press('V') await page.keyboard.up('Control') ``` You can run (from the terminal) the above examples as follows: ```sh theme={null} npx playwright test basic-click-type.ts ```

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

## Further reading 1. The related official documentation of [Playwright](https://playwright.dev/docs/input#mouse-click) 2. [Finding effective selectors](/docs/learn/playwright/selectors/) # Record Automation Scripts Using Playwright Codegen Source: https://www.checklyhq.com/docs/learn/playwright/codegen Generate Playwright scripts automatically with the codegen command, recording your browser interactions into editable test code with user-first locators. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Did you know that you can use Playwright to generate your automation scripts automatically? Let's dive into how you can leverage Playwright's `codegen` feature to record scripts effortlessly. ### Step 1: Launch Playwright Codegen To get started, you'll use the `codegen` command from the Playwright executable. Let's say you want to create a script for interacting with [Checkly](https://checklyhq.com/). You would run the following command in your terminal: ```bash theme={null} npx playwright codegen https://checklyhq.com ``` When you execute this command, two windows will open up. On the left, you have a browser window displaying the target site, and on the right, you see the Playwright inspector. Playwright codegen inspector

Note that the inspector will populate with the basic code needed for a Playwright test, and start with the line needed to load the url you gave in the terminal. Also, the browser will automatically highlight elements, and show you a locator for them in Playwright’s preferred [user-first selector](https://www.checklyhq.com/blog/playwright-user-first-selectors/) format. ### Step 2: Interact with the Website Now, you can start interacting with the website as you normally would. For example: 1. Accept cookies. 2. Click a few links. 3. Type "hello world" in a search box. As you perform these actions, you'll notice that the Playwright inspector on the right is recording every interaction in real-time. Each action is converted into a line of code, creating a script as you go. For example the steps above would produce the code as follows: ```ts theme={null} test('test', async ({ page }) => { await page.goto('https://www.checklyhq.com/'); await page.getByRole('button', { name: 'Accept All' }).click(); await page.getByRole('button', { name: 'Developers' }).first().click(); await page.getByRole('link', { name: 'Documentation Technical docs' }).click(); await page.getByLabel('Search').click(); await page.getByPlaceholder('Search docs').click(); await page.getByPlaceholder('Search docs').fill('hello world'); }); ``` Note that codegen will make a ‘best effort’ at emulating your actions and writing ‘best practice’ code, so you should always review it before deploying. ### Step 3: Add Assertions to your tests Every action in the script above has implicit assertions about the elements we interact with. Playwright will expect these elements are stable, visible, and enabled; and if one isn’t then the test will fail we could call this done, but we generally want our tests to contain assertions beyond just navigating. There are a few reasons why you should have assertions in your test: It makes it clearer to future coders what your test was checking for, and the error message is more useful from a failed assertion rather than just failing to find an element. Right now all we’ll get if the test above fails is a timeout and a line which failed. To add assertions, use the little toolbar at the top of the Chromium browser: buttons at the top of the playwright codegen browser

buttons at the top of the playwright codegen browser

* The first button, ‘Record,’ is active at startup, and you can click it to pause the inspector’s recording of your every action in the browser * Next is ‘Pick Locator,’ clicking this will make the locator for anything you click appear in the inspector, useful if you know how to write a test but are struggling to find a good locator for a particular element * The 👁️ button in the middle is ‘Assert Visible’, clicking this and clicking a page element adds an assertion to our test. for example clicking the ‘book a demo’ link in the main [checklyhq.com](https://checklyhq.com) homepage gets this assertion in our test code: `await expect(page.getByRole('button', { name: 'Book a demo' }).first()).toBeVisible();` * What if we just want to make sure that a particular string appears in an element? Click the ‘Match Text’ button and you’ll get a dialog box where to enter the text you want to check for. writing an assertion

this will generate a `.toContainText(` assertion in the inspector * Finally the ‘Assert Value’ generates a `.toHaveValue()` assertion, which is used to check that an [input field has a certain value](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-have-value), for pre-filled inputs or for example when a user’s data should automatically be filled out in a form. Note that the assertions created by codegen are fairly basic, and while they could be the whole basis for a test suite, it’s a great idea to learn about [Playwright assertions](https://www.checklyhq.com/learn/playwright/assertions/) and write more complex ones yourself. You can write any calculable value into an assertion, so the sky is the limit. ### Step 4: Write tests in Java, Python, .Net or Javascript - Choose Your Target Environment After you're done interacting with the website, you can choose the target environment for your script. Playwright allows you to generate scripts in various languages, such as JavaScript, Python, or Playwright's own test framework. For this example, let's pick JavaScript. You can copy the generated script from the inspector. ### Step 5: Save and Run Your Script With your script copied, you can drop the test into a Playwright `spec` file, and run it with `npx playwright test` to see the results. Note that the script as generated contains no comments or further explanations. If you want to be considerate of your next coder, who let’s be honest is probably you in 6 months having forgotten everything about this test, add some comments, give the test a more helpful name than `'test'` and [divide the test into steps for readability](https://www.checklyhq.com/blog/improve-your-playwright-documentation-with-steps/). ### Step 6: Run Your Script in the Cloud If you want to take it a step further, use [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) to run your Playwright tests in the Checkly infrastructure as synthetic monitoring.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

### Bonus Step 1: Add Playwright to Visual Studio Code Many developers are now comfortable using Microsoft’s Visual Studio Code as their IDE, and Playwright has a great plugin for Visual Studio Code. Install the [VS Code extension from the marketplace](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright) or from the extensions tab in VS Code. the VS code extensions tab

With the plugin installed, you can record a new test directly into your script file, and even record individual steps with ‘record at cursor.’ The plugin has a lot of other neat features including being able to run your tests right from the code editor, and show a browser running your tests. If you’re already using Visual Studio Code, or thinking about adopting it as an IDE, the Playwright plugin is worth checking out! ### Bonus Step 2: Simulate a Mobile Browser Viewport with `--viewport` Playwright can’t perfectly emulate a particular build of Mobile Opera running on a 10-year-old Android, but a good start is simulating a mobile viewport, to see if things render correctly and buttons are still visible. Start by setting your viewport size when you start up codegen: ```bash theme={null} npx playwright codegen --viewport-size=600,800 checklyhq.com ``` Your browser will start up showing the new, narrow viewport a narrow viewport

You’ll also notice in the screenshot that the inspector automatically adds a `test.use` command to set the viewport. You can go quite a bit further with these settings, altering the user-agent for your test as needed to [emulate](https://playwright.dev/docs/emulation) different devices. ## See Codegen in action To see the highly visual codegen tool in action, check out Stefan’s video below: ### Go further with Playwright You’ve just taken your first steps with creating Playwright tests. Playwright's `codegen` is a powerful tool that simplifies the process of creating automation scripts. As it produces assertions and locators that are industry standard, you should use anytime you need to make a clean, readable test or automation for your site. If you’d like to dive deeper into Playwright and Checkly’s monitoring tools, explore the rest of our [Playwright learning resources](/docs/learn/playwright/overview) to see how your end-to-end testing can grow into bullet-proof site monitoring to defend your SLA.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Playwright Debugging - Techniques for Script Troubleshooting Source: https://www.checklyhq.com/docs/learn/playwright/debugging Troubleshoot failing Playwright scripts by reading error messages in context, understanding expected behavior, and using the right tools to gain visibility. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Understanding why a script does not work as expected and finding the failure root cause are automation key skills. Given its importance and its sometimes deceptive complexity, debugging is a topic that should receive quite some attention. This article will explore basic concepts and tools to point beginners in the right direction. ## Awareness comes first Script debugging is firstly about observing and understanding. Finding out what is causing the failure (or misbehaviour) in your execution heavily depends on your knowledge of: 1. What the script you are looking at is *supposed* to do 2. How the application the script is running against is supposed to behave at each step of the script When approaching a debugging session, make sure the above points are taken care of. Skipping this step is way more likely to cost you additional time than it is to save you any. ## The error message Error messages are not present in every scenario: we might be trying to understand why a script *passes*, or why it takes longer than expected. But when we have access to an error message, we can use it to guide us. The error, in and of its own, is not always enough to understand what is going wrong with your script. Oftentimes, there can be multiple degrees of separation between the error and its root cause. For example: an ["Element not found"](/docs/learn/playwright/error-element-not-found/) error might be alerting you to the fact that an element is not being found on the page, but that itself might be because the browser was made to load the wrong URL in the first place. Do not fall into the easy trap of reading the error message and immediately jumping to conclusions. Rather, take the error message, research it if needed, combine it with your [knowledge of script and app under test](#awareness-comes-first) and treat it as the first piece to the puzzle, rather than the point of arrival of your investigation. > Good knowledge of the automation tool you are using will also help add more context to the error message itself. ## Gaining visibility Given that Headless browser scripts will run without a GUI, a visual assessment of the state of the application needs additional steps. One possibility is to adding screenshots in specific parts of the script, to validate our assumptions on what might be happening at a given moment of the execution. For example, before and after a problematic click or page transition: ```js theme={null} ... await page.screenshot({ path: 'before_click.png' }) await page.click('#button') await page.screenshot({ path: 'after_click.png' }) ... ``` Another way to better observe our script's execution is to run in headful mode: ```js theme={null} ... const browser = await chromium.launch({ headless: false, slowMo: 20 }) ... ``` We can then tweak the `slowMo` option, which adds a delay in milliseconds between interactions, to make sure the execution is not too fast for us to follow. ## Increasing logging Sometimes we need to try and see the execution through our automation tool's eyes. Added logging can help by taking us step-by-step through every command as it is executed. ```shell theme={null} DEBUG=pw:api node script.js ``` verbose playwright logs

## Accessing DevTools A wealth of information is available through the Chrome Developer Tools. We can configure our browser to start with the DevTools tab already open (this will automatically disable headless mode), which can be helpful when something is not working as expected. Careful inspection of the Console, Network and other tabs can reveal hidden errors and other important findings. debugging with chrome devtools

```js theme={null} ... await chromium.launch({ devtools: true }) ... ``` We can also use the console to directly try out a selector on the page in its current state, e.g. with `document.querySelector` or `document.querySelectorAll`. debugging selectors in browser console

If we are using Playwright, we can also run in debug mode with `PWDEBUG=console node script.js`. This provisions a `playwright` object in the browser which allows us to also try out [Playwright-specific selectors](https://playwright.dev/docs/locators). debugging playwright-specific selectors in browser console

debugging playwright-specific selectors in browser console

## The Playwright Inspector The Playwright Inspector is a GUI tool which exposes additional debugging functionality, and can be launched using `PWDEBUG=1 npm run test`. The Inspector allows us to easily step through each instruction of our script, while giving us clear information on the duration, outcome, and functioning of each. This can be helpful in [getting to the root cause](/docs/learn/playwright/debugging-errors/) of some of the more generic errors. playwright inspector debugging

> The Inspector includes additional handy features such as selector generation and debugging, as well as script recording. If a coding agent writes and fixes tests for you, all of the tools above are also available as plain text. Learn more in [Debugging with AI agents](/docs/learn/playwright/debugging-with-ai-agents/). ## Further reading 1. [Debugging challenges](/docs/learn/playwright/debugging-errors/) 2. [Working with selectors](/docs/learn/playwright/selectors/) 3. [Debugging with AI agents](/docs/learn/playwright/debugging-with-ai-agents/)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Playwright Debugging Errors - Common Issues and Solutions Source: https://www.checklyhq.com/docs/learn/playwright/debugging-errors Recognize common categories of Playwright errors and their obvious and non-obvious root causes, with links to fixes for the most frequent failures. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Helping users debug a large number of automation scripts across testing tools over many years, some high-level patterns begin to emerge. The issues that come up, even though each one might be unique, start showing their similarities and begin to appear related. This section aims to define broader categories of errors and failures in order to show the possible underlying causes and make debugging more efficient. ## Obvious vs non-obvious root causes Unfortunately, problems do not map one-to-one to solutions. If they did, debugging would be very easy. Rather, there can be more than one possible cause for an error. Sometimes the cause is obvious and sometimes it isn't, and while sometimes it is the obvious one the user should be looking at, sometimes it isn't. That is enough to create some headaches, slow down debugging and potentially whole projects with it. More often than not, beginners will be looking for a solution to the "obvious" cause, while the not-so-obvious one is actually responsible for the failure - that means they will be trying to solve the wrong problem! This is where a huge amount of time can be needlessly lost. ## Error list To help avoid stressful and unsuccessful debugging sessions, it might help to consider different possible causes depending on the error you are confronted with: 1. [Error: element not found](/docs/learn/playwright/error-element-not-found/) 2. [Error: element not visible](/docs/learn/playwright/error-element-not-visible/) 3. [Error: target closed](/docs/learn/playwright/error-target-closed/) 4. [Error: wait not respected](/docs/learn/playwright/error-wait-not-respected/) If a coding agent maintains your tests, it can run this kind of diagnosis for you: see [Debugging with AI agents](/docs/learn/playwright/debugging-with-ai-agents/).

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Debugging Playwright Tests with AI Agents Source: https://www.checklyhq.com/docs/learn/playwright/debugging-with-ai-agents Let your coding agent debug failing Playwright tests: read the error-context.md attachment, analyze traces in the terminal with npx playwright trace, and step through live tests with --debug=cli. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright's debugging tools were built for human eyes. The trace viewer, the inspector, and UI mode are great GUI tools. But when a coding agent hits a failing test, it can't click through a trace viewer. Agents prefer text. Over the last few releases, Playwright shipped text-first versions of its entire debugging toolchain. If an agent writes or fixes tests for you, these are the features it depends on. Three of them matter most: the error context attachment, the `trace` command, and `--debug=cli`. ## Failing tests ship their own context Start with the feature you get for free. When a test fails, Playwright attaches an `error-context.md` file to the test results and points to it right in the terminal output. ```txt highlight={18} theme={null} 1) [Chromium] › tests/coupon.spec.ts:3:5 › coupon code applies a discount ───────────────────────── TimeoutError: locator.fill: Timeout 3000ms exceeded. Call log: - waiting for getByLabel('Coupon code') 3 | test("coupon code applies a discount", async ({ page }) => { 4 | await page.goto("/checkout"); > 5 | await page.getByLabel("Coupon code").fill("SAVE20"); | ^ 6 | await page.getByRole("button", { name: "Apply" }).click(); 7 | await expect(page.getByText("Discount applied")).toBeVisible(); 8 | }); 9 | at /path/to/project/tests/coupon.spec.ts:5:40 Error Context: test-results/coupon-code-applies-a-discount-Chromium/error-context.md attachment #1: trace (application/zip) ───────────────────────────────────────────────────────── test-results/coupon-code-applies-a-discount-Chromium/trace.zip Usage: npx playwright show-trace test-results/coupon-code-applies-a-discount-Chromium/trace.zip ``` The file contains everything needed to understand the failure without rerunning anything: the error details, the test source, and the page snapshot at failure time. The snapshot is the page's accessibility tree rendered as YAML, the same representation agents use to "see" pages everywhere in the Playwright ecosystem. ````md theme={null} # Instructions - Following Playwright test failed. - Explain why, be concise, respect Playwright best practices. - Provide a snippet of code with the fix, if possible. # Test info - Name: coupon.spec.ts >> coupon code applies a discount - Location: tests/coupon.spec.ts:3:5 # Error details ``` TimeoutError: locator.fill: Timeout 3000ms exceeded. Call log: - waiting for getByLabel('Coupon code') ``` # Page snapshot ```yaml - main [ref=e2]: - heading "Checkout" [level=1] [ref=e3] - generic [ref=e4]: - text: Discount code - textbox "Discount code" [ref=e5] - button "Apply" [ref=e6] ``` # Test source ```ts 3 | test("coupon code applies a discount", async ({ page }) => { 4 | await page.goto("/checkout"); > 5 | await page.getByLabel("Coupon code").fill("SAVE20"); | ^ TimeoutError: locator.fill: Timeout 3000ms exceeded. 6 | await page.getByRole("button", { name: "Apply" }).click(); 7 | await expect(page.getByText("Discount applied")).toBeVisible(); 8 | }); ``` ```` Look at this example: the test waits for a `Coupon code` field, but the snapshot shows the page renders a `Discount code` textbox. The root cause is sitting right there in plain text. An agent (or you) can spot the mismatch without opening a browser. The file even starts with LLM instructions ("Following Playwright test failed. Explain why, be concise..."), so it doubles as a ready-made prompt. That's also what powers the "Copy prompt" button you'll find next to errors in the HTML report, the trace viewer and UI mode. One click, paste it into your AI chat of choice, and you're debugging. If your agent has access to your project, it will usually read `error-context.md` on its own after a failed test run. This file is a big part of why a plain "please fix this failing test" prompt works as well as it does these days. ## `npx playwright trace`: the trace viewer for terminals Playwright traces are the most complete record of an end-to-end test failure. The trace viewer makes them easy to inspect, but it's useless to an agent because it's a GUI. Prefer video? Watch the [`trace` command in action](https://www.youtube.com/shorts/gKNwkkEgcNk) in under a minute. Since [Playwright 1.59](https://playwright.dev/docs/release-notes#-cli-trace-analysis-for-agents) there's `npx playwright trace`, a command that makes recorded traces accessible from the terminal. Point it at a trace file first: ```bash theme={null} npx playwright trace open test-results/coupon-code-applies-a-discount-Chromium/trace.zip ``` Then dig in. `trace actions` lists every step of the test with timing and its failure state. ```text theme={null} # Time Action Duration ---- -------- -------------------------- -------- 8. 0:01.380 Navigate to "/checkout" 17ms 9. 0:01.398 Fill "SAVE20" 3.0s ✗ getByLabel('Coupon code') 10. 0:04.404 After Hooks 29ms ``` Every action has an index, so `trace action 9` prints the details of the failing step (parameters, error, call log), and `trace snapshot 9` renders the page snapshot exactly as it looked at that moment. `trace requests` lists the network traffic, `trace console` the console messages, and `trace errors` the collected errors with stack traces. You get the idea. It's all plain text. You can pipe it through grep or any other shell tool, and so can your agent. When a test fails, a simple prompt like this sends your agent through the entire trace, no browser needed: ```text theme={null} The test broke. Please investigate the trace! ``` To teach your agent the trace tooling, install the matching agent skill with `npx playwright trace install-skill`. After that, it knows all the commands above and when to reach for them. ## `--debug=cli`: a debugger your agent can drive Sometimes a recorded trace isn't enough and you want to poke at the live page. For humans, that's what `npx playwright test --debug` is for. It opens the Playwright inspector and lets you step through your test action by action. Your agent can't click through an inspector window, though. The Playwright inspector paused on a login test, highlighting the current action while the browser shows the page under test

The Playwright inspector paused on a login test, highlighting the current action while the browser shows the page under test

Prefer video? Watch [`--debug=cli` in action](https://www.youtube.com/shorts/qrFcZDsTOuA) in under a minute. Since [Playwright 1.59](https://playwright.dev/docs/release-notes#-cli-debugger-for-agents), the `--debug` flag accepts a `cli` mode: ```bash theme={null} npx playwright test tests/coupon.spec.ts --debug=cli ``` Instead of opening a GUI, the test run pauses and prints instructions on how to control it from another terminal session: ```text theme={null} Running 1 test using 1 worker ### The test is currently paused at the start ### Debugging Instructions - Run "npx playwright-cli attach tw-10b692" to attach to this test ``` Now attach with the Playwright CLI, and everything you'd normally do in the inspector works as plain terminal commands: ```bash theme={null} npx playwright-cli attach tw-10b692 # step to the next test action npx playwright-cli --s=tw-10b692 step-over # print the current accessibility tree npx playwright-cli --s=tw-10b692 snapshot # list the network traffic npx playwright-cli --s=tw-10b692 requests ``` Each `step-over` reports where the test is paused and what action comes next. `snapshot` shows the live page state, `requests` reveals what's happening on the network, and you can interact with the paused page using all the other CLI commands (`click`, `fill`, `eval`, and friends). **It's the same Playwright debugger, just in plain text.** For humans, this isn't a great way to work. But hand it to an agent with the Playwright CLI skill installed: ```text theme={null} The coupon test broke. Please run Playwright with the `--debug=cli` flag and debug it with the playwright-cli tooling. ``` The agent spins up a debugging session the same way you would: it steps to the failing action, checks the snapshot, inspects `requests`, and discovers, say, that the coupon HTTP call came back with a 500. Watching an agent debug via text is amazingly nerdy, and it works. Add `--headed` to the test command if you want to watch the browser while the agent drives. This workflow only works if the Playwright CLI skill is installed and up to date. Run `npx playwright-cli install` to set it up, and re-run it after updating Playwright so your agent knows the latest commands. ## Debugging doesn't stop at your test suite If you're not on a recent Playwright version, updating is probably the easiest upgrade your agent can get. And this workflow extends past pre-production testing. If you run your Playwright tests as [production monitors on Checkly](/docs/detect/synthetic-monitoring/playwright-checks/overview), the same [traces](/docs/guides/reading-traces/) get recorded on every failed check run, and [Rocky AI](/docs/resolve/ai-root-cause-analysis/overview) analyzes them for you. When a check fails, the root cause analysis is usually done before you even start digging. ## Further reading 1. [Debugging scripts](/docs/learn/playwright/debugging/) 2. [Common debugging errors](/docs/learn/playwright/debugging-errors/) 3. [Control your monitoring infrastructure with AI agents](/docs/guides/agentic-workflows/) 4. [Playwright's trace viewer docs](https://playwright.dev/docs/trace-viewer)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to emulate mobile devices with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/emulating-mobile-devices Emulate mobile devices in Playwright by setting viewport size, device pixel ratio, and user-agent strings to test responsive behavior across screens. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright controls headless desktop browsers that can also emulate mobile devices. And while device emulation can't replace testing on mobile devices entirely, it's a practical and quick-to-setup approach to testing mobile scenarios. Device emulation is well suited to test if your site behaves correctly across multiple viewport sizes and correctly handles `user-agent` strings. But if your site relies on device-specific browser features, an iPhone emulation running in a Chromium browser might lead to false positives. This guide explains how to define viewport sizes, device pixel ratio and `user-agent` strings using Playwright. ## Defining the user agent string If your site parses the user agent string to serve a different experience to mobile users, define the `userAgent` in your automation scripts. ```ts emulate-mobile.spec.ts theme={null} import { test } from '@playwright/test' test.use({ userAgent: 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/16.0 Mobile/14E304 Safari/602.1', }) test('emulate iPhone SE', async ({ page }) => { await page.goto("https://danube-web.shop/") // perform your tests }) ``` ## Defining viewport size and pixel density If your site follows responsive web design practices and renders elements depending on device viewport size, define a mobile viewport and pixel density. ```ts emulate-mobile-viewport.spec.ts theme={null} const { test } = require('@playwright/test') test.use({ viewport: { width: 320, height: 568 }, deviceScaleFactor: 2 }) test('emulate iPhone SE', async ({ page }) => { await page.goto('https://danube-web.shop/') // perform your tests }) ``` ### Use built-in device registries Playwright includes a built-in device registry to access mobile device characteristics quickly. * [Playwright devices](https://playwright.dev/docs/emulation#devices) Leverage the pre-defined devices to emulate mobile devices. ```ts emulate-mobile-builtin.spec.ts theme={null} import { test, devices } from '@playwright/test' const iPhone = devices['iPhone SE'] test.use({ ...iPhone, }) test('emulate iPhone SE', async ({ page }) => { await page.goto('https://danube-web.shop/') // perform your tests }) ``` ## Further reading 1. [Measuring page performance](/docs/learn/playwright/performance/) 2. [Playwright's emulation documentation](https://playwright.dev/docs/emulation)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Fix the "Click Not Executed" Error in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/error-click-not-executed Diagnose why a Playwright click seems to be ignored, from duplicate element IDs to the wrong element receiving the click, and use strict mode to catch it. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. In certain situations, it might look as if no click is happening in the browser even if our script specifies it. For example: our Playwright script is supposed to run a `page.click('#btn-login')` but seems to ignore the click and just proceed with the next instruction. This can result in an `Element not found error` or similar. ## Possible causes * **Not-so-obvious:** the element we are trying to click is on the page, but is not the one receiving the click; there might be another element somewhere else on the page that is receiving it instead. The instruction itself does not raise any error, as it is in fact being executed correctly. ## How to avoid confusion Try querying for the element in the browser console during inspection. If a `document.querySelectorAll('mySelector')` (or simply `$$('mySelector')`) returns more than one element, you want to come up with a more precise selector which references only the specific element you are looking to click. Unless you know for certain, do not assume that the page you are automating follows best practices. For example: IDs are unique in valid HTML, but a page can be made up of invalid HTML and still work! So if you are struggling with a click seemingly not going through and your selector is based on an ID, check whether the page contains duplicated IDs. The Chrome DevTools console will also alert you: chrome devtools console showing alert for duplicated id

chrome devtools console showing alert for duplicated id

If you are running Playwright 1.14 or newer, you can also enable [strict mode](https://playwright.dev/docs/release-notes#version-114) to have it throw an error in case your selector matches more than one element on the page: `await page.click('mySelector', { strict: true });` > Note that this list neither is nor aims to be complete: additional possible causes most likely exist for this error.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Fix 'Element Not Found' Errors in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/error-element-not-found Fix Playwright's element not found error, from wrong selectors and missing waits to a previous click that silently left you on the wrong page. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. One of perhaps the most common and direct error messages one will see, especially when starting out with writing automation scripts, will be the `Element not found` error. A variety of root causes including wrong selectors, missing waits, navigation problems and more can hide behind it. Example error message: ```sh theme={null} UnhandledPromiseRejectionWarning: Error: No node found for selector: ... ``` ## Possible causes * **Obvious possible cause #1:** the selector is wrong. See [working with selectors](/docs/learn/playwright/selectors/). * **Obvious possible cause #2:** the element is not on the page and the automation tool is not automatically waiting for it to appear. An [explicit wait](/docs/learn/playwright/navigation/) might fix the problem. * **Not-so-obvious possible cause:** the click on the previous element [did not actually go through](/docs/learn/playwright/error-click-not-executed/). From the perspective of our automation tool, everything went fine, but from ours what happened is more similar to a silent failure. We are now looking for the right element but are on the wrong page (or the page is in the wrong state), and the target element is therefore not found. ## How to avoid confusion Either walk through the execution in headful mode or take screenshots before and after the instruction that has raised the error - this will help you verify whether the application state actually is the one you expect. > Note that this list neither is nor aims to be complete: additional possible causes most likely exist for this error.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Fix "Element Not Visible" Error in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/error-element-not-visible Fix Playwright's element not visible error by checking whether an element is hidden or covered by another, using headful mode and screenshots to confirm state. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Knowing that an element is included in the DOM might not be enough for us to properly interact with it: its state also determines whether our action will be able to go through. Example error message: ```sh theme={null} ERROR - UserScript page.waitForSelector: Timeout 30000ms exceeded. waiting for selector ".contact-form > .form-control" to be visible selector resolved to hidden ``` ## Possible causes * **Obvious possible cause:** the element is set to hidden while it shouldn't. Something is wrong with the element itself. * **Not-so-obvious possible cause:** a different element is hiding the target element without our knowledge. ## How to avoid confusion Either walk through the execution in headful mode or take screenshots before and after the instruction that has raised the error - this will help you verify whether the application state actually is the one you expect. > Note that this list neither is nor aims to be complete: additional possible causes most likely exist for this error.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Fix "Target Closed" Error in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/error-target-closed Understand and fix Playwright's Target closed error, usually caused by a browser, context, or tab closing too early or a missing await on a promise. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Our script can return the sometimes cryptic "Target closed" error. Knowing that the "target" is normally our browser, the [context](https://playwright.dev/docs/browser-contexts) or tab that our script is controlling can help us close in on the cause of the issue. Example error message: ```sh theme={null} UnhandledPromiseRejectionWarning: Error: Protocol error: Target closed ``` ## Possible causes * **Obvious possible cause:** the browser, context or tab is being closed at the wrong time in the script. * **Not-so-obvious possible cause:** promises are not being handled correctly, e.g.: 1. Forgetting an `await` so that the `browser.close()` executes before your command has terminated. 2. [Wrong foreach usage](https://github.com/babel/babel/issues/909): [forEach](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach) expects a synchronous function, so use a [for-of loop](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of) if you are using `await`. > Note that this list neither is nor aims to be complete: additional possible causes most likely exist for this error.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Fix "Wait Not Respected" Error in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/error-wait-not-respected Fix Playwright waits that seem ignored, usually because the element already matches your state requirement, and confirm element state in the browser console. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Our script might sometimes look as if it were not honoring the waits we are specifying, instead proceeding and potentially running into an error on the next instruction. For example: we are waiting for an element, e.g. with `page.waitForSelector`, but the script immediately proceeds to the next instruction. This can result in an [element not found error](#element-not-found) or similar on the following step in our script. ## Possible causes * **Not-so-obvious possible cause:** the element we are waiting for is already in the DOM, possibly not visible in our inspection but already matching our `state` requirements. ## How to avoid confusion Try querying for the element in the browser console during inspection. If the element is found, inspect its attributes (e.g. `visibility`) and ensure they match your expectations. > Note that this list neither is nor aims to be complete: additional possible causes most likely exist for this error.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Download Files with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/file-download Download files with Playwright and verify them against a fixture, using either the download event or extracting the href and fetching it with an HTTP request. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Websites might expose files for users to download and then access from their local machine. Common cases are downloading tickets, receipts and itineraries. ## Steps This example runs against our [test webshop](https://danube-web.shop/) and proceeds to download a receipt for a previous purchase. It includes the following steps: 1. Logging in to the website 2. Navigating to the account page 3. Downloading a linked file We will check that the downloaded file is as expected by comparing it to a [fixture file](/docs/learn/playwright/handling-test-data/) in our final assertion. We can approach this scenario in different ways. One possibility is to perform the first two steps, then [extract](/docs/learn/playwright/web-scraping/) the `href` value and use it to retrieve the file with a `GET` request (performed with [axios](https://github.com/axios/axios), for example). ```ts title="file-download.spec.ts" theme={null} // Example code for file download using HTTP request import { test, expect } from '@playwright/test'; import axios from 'axios'; import fs from 'fs'; test('download file with HTTP request', async ({ page }) => { // Login to the website await page.goto('https://danube-web.shop/login'); await page.fill('#email', process.env.USER_EMAIL!); await page.fill('#password', process.env.USER_PASSWORD!); await page.click('#login-button'); // Navigate to account page await page.goto('https://danube-web.shop/account'); // Extract download link const downloadLink = await page.getAttribute('a[href*="receipt"]', 'href'); // Download file using HTTP request const response = await axios.get(downloadLink, { responseType: 'stream' }); const downloadPath = './downloaded-receipt.pdf'; const writer = fs.createWriteStream(downloadPath); response.data.pipe(writer); await new Promise((resolve, reject) => { writer.on('finish', resolve); writer.on('error', reject); }); // Compare with fixture file const downloadedFile = fs.readFileSync(downloadPath); const fixtureFile = fs.readFileSync('./fixtures/expected-receipt.pdf'); expect(downloadedFile).toEqual(fixtureFile); }); ``` We could also click the link directly and wait for the download event, then proceed with the comparison. Note that in this case, we need to enable downloads in the browser context before proceeding. ```ts title="file-download-alt.spec.ts" theme={null} // Example code for file download using Playwright's download event import { test, expect } from '@playwright/test'; import fs from 'fs'; test('download file with download event', async ({ page }) => { // Login to the website await page.goto('https://danube-web.shop/login'); await page.fill('#email', process.env.USER_EMAIL!); await page.fill('#password', process.env.USER_PASSWORD!); await page.click('#login-button'); // Navigate to account page await page.goto('https://danube-web.shop/account'); // Wait for download event when clicking the link const downloadPromise = page.waitForEvent('download'); await page.click('a[href*="receipt"]'); const download = await downloadPromise; // Save the downloaded file const downloadPath = './downloaded-receipt.pdf'; await download.saveAs(downloadPath); // Compare with fixture file const downloadedFile = fs.readFileSync(downloadPath); const fixtureFile = fs.readFileSync('./fixtures/expected-receipt.pdf'); expect(downloadedFile).toEqual(fixtureFile); }); ``` Both examples can be run as follows: **macOS/Linux:** ```bash theme={null} USER_EMAIL=user@email.com USER_PASSWORD=supersecure1 npx playwright test file-download.spec.ts ``` **Windows:** ```bash theme={null} SET USER_EMAIL=user@email.com SET USER_PASSWORD=supersecure1 npx playwright test file-download.spec.ts ``` ## Takeaways 1. Use environment variables to inject secrets. 2. Compare the expected file with the newly downloaded one. 3. There is more than one way to download a file within our script. ## Further reading 1. [Playwright's](https://playwright.dev/#version=v1.3.0\&path=docs%2Fapi.md\&q=class-download) documentation on downloading files.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Generate PDFs with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/generating-pdfs Generate PDFs from web pages with Playwright's page.pdf() command and customize the output for archiving, invoices, and other document automation. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright can be used to create PDFs from webpages. This opens up interesting automation scenarios for tasks such as archiving, generating invoices, writing manuals, books and more. This article introduces this functionality and shows how we can customise the PDF to fit our needs. ## Generating a PDF file After loading a page, we use the `page.pdf()` command to convert it to a PDF. ```ts title="pdf-minimal.spec.ts" theme={null} // Example code for generating a minimal PDF import { test, expect } from '@playwright/test'; test('generate PDF', async ({ page }) => { await page.goto('https://example.com'); await page.pdf({ path: 'example.pdf' }); }); ``` Note that we need to pass the `path` option to have the PDF file actually saved to disk. This feature is currently only supported in Chromium headless in Playwright. ## Tweaking the result It is important to take a quick look at the official docs for `page.pdf()` from [Playwright](https://playwright.dev/docs/api/class-page#page-pdf), as it is almost certain that we will want to tweak the appearance of our page in the resulting PDF. In certain cases, our webpage might look significantly different in our PDF compared to our browser. Depending on the case, it can pay off to experiment with the following: 1. We might need to set option `printBackground` to true in case graphical components appear to be missing in the generated PDF. 2. By default, `page.pdf()` will generate a PDF with adjusted colors for printing. Setting the CSS property `-webkit-print-color-adjust: exact` will force rendering of the original colors. 3. Calling `page.emulateMedia('screen')` changes the CSS media type of the page. 4. Setting either `width` and `height` or `format` to the appropriate value might be needed for the page to be displayed optimally. ## Customising header and footer We can also have custom headers and footers added to our pages, displaying values such as title, page number and more. Let's see how this looks on your [favourite website](https://www.checklyhq.com/): ```ts title="pdf-hd.spec.ts" theme={null} // Example code for generating a PDF with header and footer import { test, expect } from '@playwright/test'; test('generate PDF with header and footer', async ({ page }) => { await page.goto('https://www.checklyhq.com/docs/'); await page.pdf({ path: 'checkly-with-header-footer.pdf', format: 'A4', displayHeaderFooter: true, headerTemplate: '

Custom Header

', footerTemplate: '

Custom Footer

', margin: { top: '100px', bottom: '100px', left: '50px', right: '50px' } }); }); ``` Change `headerTemplate` and `footerTemplate` to include more complex templates like these ones below. **template-header.html** ```html theme={null}

``` **template-footer.html** ```html theme={null}

Page of

``` Then, the first page of the generated PDF looks as follows: generated pdf example

Run the above examples as follows: ```bash theme={null} npx playwright test generate-pdf.ts ``` ## Further considerations We can easily transform existing web pages into PDF format, just as we have shown in our example. An even more interesting use case is about generating a brand new document: now we can use our existing HTML and CSS skills to produce high-quality PDFs, often eliminating the need for LaTeX or similar tools.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Automate Google Login with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/google-login-automation Automate social login with a Google account in Playwright, injecting credentials through environment variables to sign in via an OAuth provider flow. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Social login using your personal Google or Google Gsuite account is a common use case for many login scenarios. ## Steps 1. We start at a site that offers Google as an authentication provider. In this case we use [Stack Overflow](https://stackoverflow.com/). 2. We fetch the login page and click the "Login with Google" button. 3. We are redirect to Google. 4. We provide the username and password, injected by using environment variables. 5. We are redirected back to the starting. ```ts google-login.spec.ts theme={null} import { test } from '@playwright/test' test('Google login', async ({ page }) => { await page.goto('https://stackoverflow.com/users/login') await page.getByRole('button', { name: 'Log in with Google' }).click() await page.getByLabel('Email or phone').fill(process.env.GOOGLE_USER) await page.getByRole('button', { name: 'Next' }).click() await page.getByLabel('Enter your password').fill(process.env.GOOGLE_PWD) await page.getByRole('button', { name: 'Next' }).click() await page.getByRole('button', { name: 'Continue' }).click() }) ``` Run this example as follows. Replace the username and password placeholder with your own credentials. ```sh theme={null} GOOGLE_USER=username GOOGLE_PWD=password npx playwright test google-login.spec.ts ``` ```sh theme={null} SET GOOGLE_USER=username SET GOOGLE_PWD=password npx playwright test google-login.spec.ts ``` This example does not work when you have 2-factor authentication enabled, and you might trigger a recaptcha check. ## Takeaways 1. Use environment variables to inject secrets. 2. Waiting for the navigation as you are redirected to Google is done automatically. 3. Waiting for the navigation as you are redirected back to the start site is done automatically.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Handle Test Data with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/handling-test-data Keep Playwright tests DRY by factoring out test data and fixtures into dedicated files, so complex end-to-end scenarios stay readable and maintainable. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. We define *test data* as any data we consistently use to verify properties such as a system's functionality or performance. Another popular term for the same concept is *fixture*. We will use these interchangeably. ## Avoiding duplication It is often desirable to [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) up our scripts by factoring out test data. In the case of simple tests, it's usually not an issue to embed test data directly inside our script, but complex end-to-end scenarios might require moving this information elsewhere, like a dedicated file. Looking at our [test webshop](https://danube-web.shop/), we might want to verify that a specific item list is loaded on the store's front page. As this list contains several tens of elements, each with different attributes, keeping our fixtures inside our script would be impractical. Let's add this data to a JSON file instead: ```json theme={null} [ { "title": "Haben oder haben", "author": "Fric Eromm", "price": "9.95" }, { "title": "Parry Hotter", "author": "J/K Rowlin'", "price": "9.95" }, { "title": "Laughterhouse-Five", "author": "Truk Tugennov", "price": "9.95" }, { "title": "To Mock a Killingbird", "author": "Larper Hee", "price": "9.95" }, { "title": "1498", "author": "Gorge Norwell", "price": "9.95" }, { "title": "The Grand Grotsby", "author": "Gerald F. Scott", "price": "9.95" }, { "title": "The Pickled Lynx", "author": "Ant One", "price": "9.95" }, { "title": "Celsius 233", "author": "Bay Radbdury", "price": "9.95" }, { "title": "The Rye in the Catcher", "author": "DJ Salinger", "price": "9.95" }, { "title": "Of Mouse and Man", "author": "Johannes Beckstein", "price": "9.95" }, ... ] ``` We are then able to feed this file into our test... ```js theme={null} const fs = require('fs') ... let rawdata = fs.readFileSync('books.json') const bookList = JSON.parse(rawdata) const foundList = bookList ``` ...and have each comparison executed to ensure the right elements are being shown. ```js theme={null} // remove every element found from the control array... for (i = 0; i < resultsNumber; i++) { const bookTitle = await page.$eval(`.preview:nth-child(${i+1}) > .preview-title`, e => e.innerText) const bookAuthor = await page.$eval(`.preview:nth-child(${i+1}) > .preview-author`, e => e.innerText) foundList = foundList.filter(e => (!((e.title === bookTitle) && (e.author === bookAuthor)))) } // ...then assert that the control array is now empty assert.equal(foundList.length, 0) ``` ## Retrieving test data If the platform you are testing exposes an API endpoint to pull up-to-date test data, you could fetch the file as part of the setup phase of your test and then utilize it: ```js theme={null} const axios = require('axios') ... const { data } = await axios.get('https://danube-web.shop/api/books') const bookList = JSON.parse(data) const foundList = bookList ``` This approach enables testing with production data, so you don't have to maintain the test data yourself. ## Generating test data If you test form submissions, relying on generated data might also make sense to avoid only testing the happy path. [Faker](https://fakerjs.dev/) is a library to create fake but realistic data sets. ```js theme={null} const { faker } = require("@faker-js/faker") ... await page.type('#s-name', faker.name.firstName()) // Leif await page.type('#s-surname', faker.name.lastName()) // Kirlin await page.type('#s-address', faker.address.streetAddress(true)) // 2629 Ross Glens Suite 089 await page.type('#s-zipcode', faker.address.zipCode()) // 03651 await page.type('#s-city', faker.address.city()) // Rohanfort await page.type('#s-company', faker.company.companyName()) // Reynolds Group ``` Faker covers a wide range of data that includes names, addresses, products, images and much more. ## Takeaways 1. Keep test data separate from your scripts. 2. REST APIs can be useful for retrieving test data. 3. Libraries such as Faker can help generate test data.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to detect broken links using Playwright Source: https://www.checklyhq.com/docs/learn/playwright/how-to-detect-broken-links Detect broken links on a page with Playwright by extracting every link and checking each URL's status code, then monitor for 404s with Checkly. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. One of our Slack community members recently asked if they could use Playwright and Checkly to detect broken links on their sites. They certainly can, and the answer to this question covers so many different Playwright concepts that it makes a perfect case for sharing Playwright features with the community. Let's unveil some links going nowhere! If you prefer the video version of this tutorial, check the video below, and if you're only here to copy and paste a final implementation, find the final code on GitHub. ## Let's break down the problem To detect broken links in an automated Playwright script, we have to do two things: * Detect and extract all links on a visited page. * Make requests to all these URLs and evaluate their status code. ```js theme={null} test(`A page has no 404s`, async ({ page }, testInfo) => { // Navigate to the page you want to check for broken links await page.goto("https://your-url-to-test.com") // 1. Extract all the link URLs from a page // 2. Evaluate the URLs and check for proper status code }) ``` Let's tackle these two tasks step by step. ## How to extract the `href` attribute from all links on a page To extract all these `href` values, you could think of calling a locator like `page.getByRole('link')` or `page.locator('a')`, iterate over all the link elements, and access the href attribute… But unfortunately, this approach won't work because you can't iterate over a locator. Playwright locators are lazy and will only be evaluated when combined with an action or assertion. ```js theme={null} // Define a locator. const cta = page.getByRole('button', {name: "Click me"}) // Evaluate the locator, reach into the DOM // and click the resulting element. await cta.click() // Evaluate the locator, reach into the DOM // and check if the matching element is visible. await expect(cta).toBeVisible() ``` This locator behavior is why we can rely on Playwright's [auto-waiting](/docs/learn/playwright/waits-and-timeouts) in end-to-end testing. You define a target element "blueprint", and when this blueprint is used with an action or assertion, Playwright will constantly query the DOM and wait for elements to match. It's a great concept! In our case of extracting links, this lazy behavior isn't helpful, though. How could you immediately reach into the DOM and evaluate all the links? ### `locator.all()` — access the DOM right away If you're going beyond the classical end-to-end testing case like we do right now, you can call the locator.all method to reach into the DOM and receive an Array of locators matching the currently present DOM elements. With this method at hand, we can evaluate all the link target URL. const links = page.locator("a") ```js theme={null} // Evaluate all available links. const allLinks = await links.all() // Wait for all the getAttribute() calls to resolve. const allLinkHrefs = await Promise.all( // Extract the `href` attribute. allLinks.map((link) => link.getAttribute("href")) ) // ['https://example.com', '/something', '/something-else', ...] ``` After resolving the locator definition with .all() we can iterate over locators and map them to their attribute value with getAttribute(). Note that getAttribute() is an async operation returning another promise, so we must wrap everything into a Promise.all to wait for all the href values to be accessible. With these few lines, we extracted all the href values. We could now evaluate if these URLs return will return a valid 200 status code when called, but let's add more features to our link extraction. ### Save requests and remove duplicate link targets When we evaluated all the links, there's a high chance that the collection includes duplicates. For example, a link to home `(/)` will probably be included multiple times. And while these duplicates aren't a big deal, why should we check a target URL for a good status code multiple times? Let's remove the duplicates by betting on a native JavaScript set. Sets have the wonderful characteristic that they only hold unique values. When we add the same value twice, it'll be automatically ignored. We don't have to check if a value is already in the set. Nice! And when we already iterate over the link targets, we can also remove `mailto:` and anchor links (`#something`) in the same go! ```js theme={null} // Transform the array of link targets to a set to avoid duplicates. const validHrefs = allLinkHrefs.reduce((links, link) => { // Filter out untruthy href, `mailto:` and `#` links. if (link && !link?.startsWith("mailto:") && !link?.startsWith("#")) links.add(link) return links }, new Set()) ``` With these lines, we removed the duplicates, but did you notice that we're now filtering out links that don't have a truthy href value, too? The reason for this is that links could also just include an empty string (``). Clicking these links will only reload the page and shouldn't be on your pages either. But if we filter these out, we won't know the page has empty links. Let's add a soft assertion to our link mapping to get notified about empty links! ### Add Playwright's soft assertions to collect errors but keep the test running Whenever you use a Playwright assertion with `expect`, these assertions will throw an exception and prevent your test case from running. For end-to-end test cases, this behavior makes sense. When you click a button, expect a modal to appear to fill out an included form; if the modal doesn't show, the form-filling Playwright instructions will also fail. So why continue the test? ```js theme={null} // This exception will throw and stop a test case. await expect(headline).toBeVisible() ``` But in our case of iterating and evaluating links, we don't want to throw failed assertions and keep the test running. Whenever we discover an invalid link target (or a bad status code later), we want to continue the test to check the remaining URLs and only fail the entire test case at the end. How could we do this? For these cases, Playwright supports the concept of soft assertions `(expect.soft())`. Soft assertions work the same way as regular ones, but they won't throw on failure. Errors will be collected and displayed at the end of your test case. ```js theme={null} const validHrefs = allLinkHrefs.reduce((links, link) => { // Evaluate link target but don't throw a failed assertion error. expect.soft(link).toBeTruthy() if (link && !link?.startsWith("mailto:") && !link?.startsWith("#")) links.add(link) return links }, new Set()) ``` So now we can collect all these empty link errors, too. There's one last piece missing! ### Normalize local link targets and guarantee absolute URLs When we extract all the `href` values, we'll likely discover local links such as `/` or `/features`. If we want to check the status code of the resulting URLs, we can't request these because they need a proper protocol and domain. To transform relative links to absolute URLs, we can use another native JavaScript goodie — the [URL() constructor](https://developer.mozilla.org/en-US/docs/Web/API/URL). I won't get into much detail here, but `URL()` is the powerhouse behind most JavaScript URL operations. You can pass it a URL (it doesn't matter if it's relative or absolute), and a base URL and `new URL()` will do all the URL parsing for you. It's pretty darn sweet! ```js theme={null} new URL( "https://checklyhq.com", "https://example.com" ).href // 👆 "https://checklyhq.com" new URL( "/raccoon", "https://checklyhq.com" ).href // 👆 "https://checklyhq.com/raccoon" new URL( "/raccoon", "https://checklyhq.com/some-path" ).href // 👆 "https://checklyhq.com/raccoon" ``` With this knowledge we can tweak our link extracting a final time to ensure all found link target URLs will be absolute. ```js theme={null} const validHrefs = allLinkHrefs.reduce((links, link) => { expect.soft(link).toBeTruthy() if (link && !link?.startsWith("mailto:") && !link?.startsWith("#")) // Ensure that all URLs are absolute. links.add(new URL(link, page.url()).href) return links }, new Set()) ``` By calling `new URL()` with the extracted link and the current page URL (`page.url().href`), we can normalize all link targets. And now we're ready to check if all the URLs return a proper status code! If you're looking for the final snippet to extract link target URLs, find it on GitHub. ## How to check for broken link URLs Now that we have a set holding all the URLs, we can start making requests and check for green status codes. We could reach for Playwright's `request` fixture, but luckily, the page object also holds a request object for us. But what's the difference between the two? `page.request` will make requests in the context of the current page. For example, if you have a test case that logs in a user, the current `page` object will hold some session cookies. And if you then make requests with `page.request`, the HTTP call will include these session cookies, too. Whenever you want to make API calls on behalf of a logged-in user, `page.request` is the way to go! ```js theme={null} // Make a request using the current page's session data. const response = await page.request.get(url) From here, we can iterate over all the request URLs and check if they all return a green status code. // Iterate over the URLs and check the status code. for (const url of validHrefs) { try { const response = await page.request.get(url) expect .soft(response.ok(), `${url} has no green status code`) .toBeTruthy() } catch { expect.soft(null, `${url} has no green status code`).toBeTruthy() } } ``` The loop includes soft assertions to keep running on failure, and we added custom error messages for easier debugging. And that's pretty much it! You can now run Playwright in your CICD pipeline and check for broken links whenever you deploy your sites. Success! Again, you can find the final snippet on [GitHub](https://github.com/checkly/playwright-examples/tree/main/404-detection).

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Parameterizing your Playwright projects Source: https://www.checklyhq.com/docs/learn/playwright/how-to-parameterize-playwright-projects Pass configuration options to Playwright fixtures and page object models, so you can parameterize page objects and your entire project cleanly. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. In a previously [released YouTube video](https://www.youtube.com/watch?v=k488kAtT-Pw), I explained how and why Playwright fixtures perfectly match with page object models. Combining the two allows you to hide setup instructions and keep your tests clean. Page object models no longer have to be initialized in every test case. ```js theme={null} // Fixtures enable you to hide setup instructions from your test cases test("feature works", async ({ yourPageObjectModel }) => { /* … */ }) ``` To be upfront — I'm a fixture fanboy! But what if you need to pass additional configuration to your page object models? When options are hidden in a fixture, you can't configure how a class is initialized, right? Wrong! Let me show in this article how to implement options to configure fixtures, page object models, and even your entire Playwright code base. If you prefer video, here's a recording explaining the same concepts. ## Why fixtures enable clean Playwright projects But let's take a step back first... Over time, more and more people have asked me how they should organize and structure complex Playwright projects. I don't believe there's a single silver bullet, however, I do like to establish some patterns that enable coding conventions and best practices. As mentioned above, one of these patterns is to rely on [custom Playwright fixtures](https://playwright.dev/docs/test-fixtures#creating-a-fixture) whenever possible. As the Playwright docs state, fixtures: * help to encapsulate setup and teardown in a single place. * are reusable between tests. * are lazy and only run when "requested" in your test cases. * are composable to group functionality. And I wholeheartedly agree with all these statements, but let me add one more point for using fixtures: implementing custom fixtures "just feels right"™. By using fixtures, you're extending Playwright's core functionality rather than creating a spaghetti code base with tests importing JS/TS modules from anywhere. **Fixtures allow you to follow Playwright's design principles; you'll be using the testing framework as it's supposed to!** Let's look at a fixture example that hides the initialization of a page object model. ```ts theme={null} // BEFORE import { DashboardPage } from './poms/dashboard-page.ts' test('create check', async ({ page }) => { const dashboardPage = new DashboardPage(page, { email: 'stefan@checklyhq.com', password: '...' }); await dashboardPage.login() await dashboardPage.createCheck() }) // AFTER test('create check', async ({ dashboardPage }) => { await dashboardPage.createCheck() }) ``` By moving the initialization of `DashboardPage` into a fixture and Playwright itself, the test cases can focus on testing features. You don't need to repeatedly add setup instructions that initialize page object models. Define that you want to use a dashboardPage in your test and receive a ready-to-use object. Great stuff! But if your page object models need additional configuration, such as the user above, you might wonder how to pass additional arguments to your page object model constructors. Let's look at how this fixture is implemented and find out how we can configure it! ## How to add options to your custom Playwright fixtures Suppose you haven't used fixtures before; here's all the magic. Instead of importing test and expect from @playwright/test, your tests can import these from a base file that extends Playwright's core functionality. ```ts base.ts theme={null} // 1. import `test` but rename it to `base` import { test as base } from '@playwright/test' import { DashboardPage, User } from './poms/dashboard' // 2. extend `base` with your custom functionality // 3. export and use the returned value as `test` export const test = base.extend({ dashboardPage: async ({ page }, use) => { const dashboardPage = new DashboardPage(page, { email: 'stefan@checklyhq.com', Password: '...' }) await dashboardPage.login() // whatever you call `use` with will be passed to the // tests requesting to use the `dashboardPage` fixture await use(dashboardPage) }, }) export { expect } from '@playwright/test' ``` Playwright's `test` method allows you to enrich Playwright's core functionality with your own custom fixtures and options. The trick is to import and rename `test`, extend it with your custom additions, and then export the returned value as `test` to be used in your test cases. In the example above, the `dashboard` fixture initializes the new `DashboardPage` and then passes the result to the provided `use` method. Whatever `use` will be called with will be passed to your test cases. This has the advantage of having all your setup instructions in one place. But if you now have test cases that should use the `dashboardPage` object with different users, you can't change how the page object model will be initialized because the setup is hidden away from the test cases. How could we make this fixture setup configurable? ## Configure your Playwright project with custom options Similar to fixtures, Playwright allows you to define static values and options in our Playwright setup, and you can do it right next to your fixture definitions in the extend call. ```ts base.ts theme={null} import { test as base } from '@playwright/test' import { DashboardPage} from './poms/dashboard' export const test = base.extend({ // define a new Playwright option user: [ { email: 'stefan@checklyhq.com', password: '...' }, { option: true }, ], dashboardPage: async ({ page }, use) => { /* ... */ }, }) export { expect } from '@playwright/test' ``` Now that you have defined a user in your new test object, tests can access this option. ```ts theme={null} test('create check', async ({ dashboardPage, user }) => { // do something with the new `user` option }) ``` But it's not only your tests that can access this newly defined user; your fixtures can access it, too! ```ts base.ts theme={null} import { test as base } from '@playwright/test' import { DashboardPage} from './poms/dashboard' export const test = base.extend({ user: [ { email: 'stefan@checklyhq.com', password: '...' }, { option: true }, ], dashboardPage: async ({ page, user }, use) => { // use the new `user` object in your fixture const dashboardPage = new DashboardPage(page, user) // ... }, }) export { expect } from '@playwright/test' ``` And this is great, but so far we only shuffled some code around, we didn't find a solution to change the user. Did you notice the option: true? By marking the Playwright option as configurable, you can now change it from multiple places. And it will behave similarly to other "official" Playwright options. For example, you could now head into your `playwright.config.ts` to change the user object and set a different one in the global settings. ```ts playwright.config.ts theme={null} import { defineConfig, devices } from '@playwright/test' export default defineConfig({ // more Playwright config here ... use: { // override your user in the global config user: { email: 'stefan@checklyhq.com', password: '...' } }, }) ``` Or, if you want to leverage Playwright projects for different users, you could do that, too! ```ts playwright.config.ts theme={null} import { defineConfig, devices } from '@playwright/test' export default defineConfig({ // more Playwright config here ... projects: [ { name: 'user-a', use: { ...devices['Desktop Chrome'], // override the default user in a project user: { email: 'stefan@checklyhq.com', password: '...' }, }, }, ], }) ``` And for special occasions, if you only have one test requiring another user, you could even change it right in your test case with `test.use`. ```ts base.ts theme={null} import { test } from './base' // override the default user test.use({ user: { email: 'hello@checklyhq.com', password: '' } }) test('create check', async ({ dashboardPage }) => { // dashboardPage now uses the user defined above await dashboardPage.createCheck() }) ``` Either way, tests or fixtures can access your user config option. And if you want to change it, you can do it similarly to other Playwright settings with the already provided use method. Win-win! ## Conclusion If your Playwright project grows, I highly recommend looking into ways to establish conventions and keep your tests clean. Fixtures and options are the perfect tool for this. Setting these up might feel like overkill, but trust me, you won't look back once you define your dependencies in this Playwright-native way. If you have any questions or comments, say "Hi" in the Checkly community Slack! But what's Checkly? Checkly enables you to take your existing Playwright tests running in CI/CD and transform them into synthetic [end-to-end monitoring](https://www.checklyhq.com/guides/end-to-end-monitoring/). Your Playwright tests will run around the clock and from anywhere on this planet so that you know that your production sites are up and running. And if they're not, you'll be the first to know because Checkly will alert you. It's pretty cool, trust me. 😉

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to search with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/how-to-search Test a site's search function end to end with Playwright by entering a term and asserting the expected number and content of results are shown. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Searching a website is an everyday action for most internet users. For most services, the speed at which their customers are able to get to the products they are looking for is directly tied to revenue. To enable that, a performant and reliable search function is needed. ## Steps The example below, which is running against our [test webshop](https://danube-web.shop/), shows how we can verify the correctness of a search's result from an end-to-end perspective. In short, we will: 1. Enter a known search term 2. Firstly, assert the expected number of results is being shown 3. If the previous point is true, assert that all expected search results are shown ```ts title="search.spec.ts" theme={null} // Example code for testing search functionality import { test, expect } from '@playwright/test'; test('search functionality', async ({ page }) => { await page.goto('https://danube-web.shop/'); // Enter search term await page.fill('#search-input', 'book'); await page.click('#search-button'); // Assert expected number of results await expect(page.locator('.search-result')).toHaveCount(4); // Assert all expected search results are shown await expect(page.locator('.search-result')).toContainText([ 'Haben oder haben', 'Parry Hotter', 'Laughterhouse-Five', 'To Mock a Killingbird' ]); }); ``` Run this example as follows: ```sh theme={null} npx playwright test search.spec.ts ``` When testing search on large sets of data, you might additionally need to handle result pagination, together with the possibility of duplicate results. Listing search terms and their corresponding expected results in a file could be helpful for additional, more thorough testing. The contents of the file would be then used to drive the searches and comparisons. An example could look like the following JSON: ```json theme={null} [ { "search": "pen", "results": [ "red pen", "blue pen", "fountain pen" ] }, { "search": "pencil", "results": [ "short pencil", "pencil sharpener", "pencil case" ] } ] ``` ## Takeaways 1. Moving test data to a separate file can help when running more thorough comparisons. 2. Assertion libraries help us cleanly verify multiple constraints in our test.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Set Up Playwright Locally Source: https://www.checklyhq.com/docs/learn/playwright/how-to-set-up-locally Install Playwright locally with a single npm command and run your first browser automation script, using Node.js and the bundled browser. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Let's start by creating a new directory and navigating to it. Assuming you already have [Node.js](https://nodejs.org/) available in your local environment, installing Playwright is achieved with just one instruction: ```sh install-playwright theme={null} npm init @playwright/test ``` Playwright comes bundled with a connected browser, so we now have all we need to run our first script. Let's create a script to navigate to our [test website](https://danube-web.shop/): ```ts basic-navigation.spec.ts theme={null} import { test } from '@playwright/test' test('basic navigation', async ({ page }) => { await page.goto('https://danube-web.shop/') }) ``` Run this example as follows: ```sh run-test theme={null} npx playwright test basic-navigation.spec.ts ``` Nothing much has happened, right? Remember: by default, Playwright will run in headless mode! That means we won't see anything of what is happening in the browser when our script runs. > Playwright creates its own browser user profile, which it cleans up on every run. In other words: all runs will be sandboxed and not interfere with one another, as state is always fully reset at the end of a session. When you are first writing and debugging your scripts, it is a good idea to enable "headed" mode, so you can have a look at what your script is doing: ```bash run-test-headed theme={null} npx playwright test basic-navigation.spec.ts --headed ``` After executing the updated file, you will see Chromium starting up, only to shut down after an instant. Everything is working as expected! Our script is just so short, it runs almost instantaneously. ## Further reading 1. Getting started guides for [Playwright](https://playwright.dev/docs/intro#installation) # How to Handle iFrames in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/iframe-interaction Access and interact with iframe content in Playwright using frameLocator to query elements inside a frame as if you were in the page context. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright enables us to access and interact with iframes. ## Locate an iframe and its elements To access iframe elements, locate the iframe and query the DOM elements as if you're in the page context. ```ts iframe-access.spec.ts theme={null} import { test } from '@playwright/test' test('access iframe content', async ({ page }) => { await page.goto('https://your-page-with-an-iframe.com') const header = await page.frameLocator('iframe').locator('h1') console.log(await header.innerText()) }) ```

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

## Further reading 1. [Playwright's "Frames documentation"](https://playwright.dev/docs/frames) # How to Intercept Requests in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/intercept-requests Observe and manipulate network traffic in Playwright by intercepting requests and responses, logging them or modifying them during a test. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. When we browse the web, a series of HTTP requests and responses are exchanged between our browser and the pages we are visiting. There are scenarios in which it is useful to monitor or manipulate this traffic. ## Request interception Request interception enables us to observe which requests and responses are being exchanged as part of our script's execution. For example, this is how we could print them out when we load our [test website](https://danube-web.shop/): ```ts title="request-interception-read.spec.ts" theme={null} // Example code for request interception import { test, expect } from '@playwright/test'; test('intercept requests', async ({ page }) => { // Listen for all requests page.on('request', request => { console.log('Request:', request.url()); }); // Listen for all responses page.on('response', response => { console.log('Response:', response.url(), response.status()); }); await page.goto('https://danube-web.shop/'); }); ``` We might want to intervene and filter the outgoing requests. For example, when [scraping web pages](/docs/learn/playwright/web-scraping/), we might want to block unnecessary elements from loading in order to speed up the procedure and lower bandwidth usage. ```ts title="request-interception-block.spec.ts" theme={null} // Example code for blocking requests import { test, expect } from '@playwright/test'; test('block image requests', async ({ page }) => { // Block image requests await page.route('**/*', route => { if (route.request().resourceType() === 'image') { route.abort(); } else { route.continue(); } }); await page.goto('https://danube-web.shop/'); }); ``` As a result, you will see the website logo not being loaded. test site without images

Similarly, switching the `resourceType` to `stylesheet` would result in the target website loading without any CSS styling. test site without css

## Response interception Isolating one or more software components from their dependencies makes them easier to test. We can do so by substituting interactions with such dependencies with simulated, simplified ones. This is also known as *stubbing*. Playwright makes it easy for us, as for every request we can intercept we also can stub a response. Every time we load it, our test website is sending a request to its backend to fetch a list of best selling books. For our example, we are going to intercept this response and modify it to return a single book we define on the fly. ```ts title="response-interception.spec.ts" theme={null} // Example code for response interception import { test, expect } from '@playwright/test'; test('intercept and modify response', async ({ page }) => { // Intercept API calls and return custom data await page.route('**/api/best-sellers', route => { const customResponse = { books: [ { title: 'Custom Book Title', author: 'Custom Author', genre: 'Custom Genre', price: '$19.99', rating: '4.5 stars' } ] }; route.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify(customResponse) }); }); await page.goto('https://danube-web.shop/'); }); ``` Here is what the homepage will look like with our stubbed response: test site with stubbed response

Run the above examples as follows: ```bash theme={null} npx playwright test request-interception.ts ``` ## Takeaways 1. Playwright gives us control over outgoing HTTP requests. 2. Playwright can easily stub HTTP responses. ## Avoid Network Interception in Synthetic Monitoring Network interception is valuable if your primary goal is to test a frontend application in isolation. However, if you want to test your application end-to-end or [use Playwright for synthetic production monitoring with Checkly](/docs/detect/synthetic-monitoring/playwright-checks/overview) you should avoid network interception as much as possible. The goal of end-to-end testing and synthetic monitoring is to test the entire stack ranging from the backend systems to the last third-party JavaScript snippet being loaded in the frontend. * You won't learn about your image CDN's downtime if your tests block image requests to save bandwidth. * You will never learn about your flaky checkout API flow if you mock all the `/order` requests. * You won't be notified when a tracking script breaks your page layout if you block third-parties in your tests. Investing in fully end-to-end testing and monitoring can be challenging but it'll provide you the safety net you're aiming for. ## Further reading 1. Official documentation on this topic from [Playwright](https://playwright.dev/docs/network). 2. [Mocks Aren't Stubs](https://martinfowler.com/articles/mocksArentStubs.html) by Martin Fowler.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Automate Login with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/login-automation Automate a standard username-and-password login flow with Playwright and assert it succeeded by checking for an element only visible to logged-in users. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. When automating key site transactions, we inevitably stumble into login scenarios. In most cases, users need to be able to access accounts on a platform to get any value out of it. If they suddenly become unable to do so, we need to be informed as quickly as possible. ## Steps In its simplest form, a login procedure requires the user to: 1. Navigate to the login form 2. Fill in a username/email field 3. Fill in a password field 4. Click a button to finalise the login At the end of our test, we need to check if our login procedure has been successful. For example, we could verify that an element is shown that we know only appears for logged-in users. On our [test site](https://danube-web.shop/) this could look like the following: ```ts login.spec.ts theme={null} import { test, expect } from '@playwright/test' test('Login', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() await page.getByPlaceholder('Email').fill(process.env.USER_EMAIL) await page.getByPlaceholder('Password').fill(process.env.USER_PASSWORD) await page.getByRole('button', { name: 'Sign In' }).click() await expect(page.getByText('Welcome back, user@email.com')).toBeVisible() }) ``` Run this example as follows. Replace the username and password placeholder with your own credentials. **macOS/Linux:** ```bash theme={null} USER_EMAIL=user@email.com USER_PASSWORD=supersecure1 npx playwright test login.spec.ts ``` **Windows:** ```bash theme={null} SET USER_EMAIL=user@email.com SET USER_PASSWORD=supersecure1 npx playwright test login.spec.ts ``` ## Takeaways 1. Use environment variables to inject secrets. 2. You might need to wait for the navigation as you are redirected to the login screen/modal.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Automate Microsoft Live Login with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/microsoft-login-automation Automate signing in to a Microsoft Live account with Playwright, injecting credentials through environment variables to reach the account page. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright allows us to automate logging in to a Microsoft Online account. ## Steps 1. We start at `https://login.microsoftonline.com/` 2. We provide the username and password, injected by using environment variables 3. We are redirected to the main account page ```ts ms-account-login.spec.ts theme={null} import { test } from '@playwright/test' test('test', async ({ page }) => { await page.goto('https://login.microsoftonline.com/') await page.getByRole('heading', { name: 'Sign in' }).click() await page.getByPlaceholder('Email address, phone number').fill(process.env.MS_USERNAME) await page.getByRole('button', { name: 'Next' }).click() await page.getByTestId('i0118').fill(process.env.MS_PASSWORD) await page.getByTestId('textButtonContainer').getByRole('button', { name: 'Sign in' }).click() await page.getByTestId('checkboxField').check() await page.getByLabel('Stay signed in?').click() }) ``` Run this example as follows. Replace the username and password placeholder with your own credentials. ```sh theme={null} MS_USER=username MS_PWD=password npx playwright test ms-account-login.spec.ts ``` ```sh theme={null} SET MS_USER=username SET MS_PWD=password npx playwright test ms-account-login.spec.ts ``` > This example does not work when you have 2-factor authentication enabled, and you might trigger a recaptcha check. ## Takeaways 1. Use environment variables to inject secrets. 2. Wait for the navigation as you are redirected to Microsoft. 3. Wait for the navigation as you are redirected back to the start site.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Test Dynamic Content by Mocking API Responses Source: https://www.checklyhq.com/docs/learn/playwright/mock-api Stabilize tests for dynamic pages by mocking API responses in Playwright, fixing unpredictable data like stock counts while leaving the rest of the page live. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright is built to interact with web pages the way your users would. Playwright’s flexible [selectors](https://www.checklyhq.com/learn/playwright/selectors/) and assertions mean that any automated test will dynamically find elements on the page even if they’ve slightly shifted visually. But how do you test dynamic content on your page? If the page content shifts constantly, we’d still like to be able to check that a certain version of the page is rendering correctly. In this article we’ll learn how to ‘fix’ dynamic content with API mocking, which lets you edit API responses while leaving the rest of the response dynamic. ## The problem, dynamic content on a critical page For this test we’re using the demo Danube web shop, testing a book’s detail page for the correct product information. Our e-commerce storefront

We started with a very simple test that a particular book’s detail page has all the correct information, but I ran into a problem with dynamic content: ```typescript book-details.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('Book has correct details', async ({ page }) => { await page.goto('https://danube-webshop.herokuapp.com/books/23'); //Removed for brevity: checks of the book's title, genre, etc. await expect(page.locator('#app-content')).toContainText('Left in stock: 1'); }); ``` There are some ways to ‘fake’ the other dynamic components like price or rating: perhaps I could test only records that have a very stable price and rating. But you can imagine the trouble with trying to check a stock number: it shifts constantly! It’s no surprise that this stock number is set on the page by a call to our books API. Let’s look at a few options to move forward with testing this page. ## Options for testing dynamic content Here are some of the paths we could take with dynamic information like a book’s rating or stock number on our demo site. All of these approaches are workable for some subset of dynamic content on pages we want to test, but we want to understand all four to create stable tests in every situation: 1. Don’t test the dynamic content - only test things that should remain constant like the book title. This is often our default approach, but obviously means that we miss testing big chunks of the page. 2. Populate our test assertions with an API call - if details like price are constantly shifting, it may make sense to have our test start with a request to our API, loading in details from that API, and using that response to create our assertion dynamically. One problem with this approach: if we make assertions based on the same API as the page, we’re no longer ‘testing’ that this API response looks right to us, since the response sets the requirements. For example if the API responded `stock:"potato"`, both the page and the test would just check for “Left in stock: potato” and the test would pass. 3. Make our assertion dynamic with a regular expression - right now our final assertion is `await expect(page.locator('#app-content')).toContainText('Left in stock: 1');` but the `toContainText` assertion will accept a regex, so we could write a passing test with: `await expect(page.locator('#app-content')).toContainText(/Left in stock: \d/);` Numbers other than 1 would now pass, but if the page rendered “Left in stock: potato” the test would fail. Solving this (and most other) dynamic content problems with a regex has two major drawbacks: * When writing easy-to-maintain testing, a regex should not be the tool we use first. Not every developer finds regex patterns easy to read or edit. * Not everything is particularly easy to test with regex, for example the `rating` api response here comes back as Unicode stars like “★★★★☆,” which isn’t as straightforward to test with regex as a digit. 4. Modify the API response - Finally the option that gives us high consistency in what we’re testing and what we expect in response. By modifying the API response we can set only the key value pairs that concern us to fixed values, letting the others remain dynamic. We test how the page renders with fixed data, *and* we’re still testing that the API is available. ## The Solution: Modify an API response with `await *page*.route()` Let’s start by looking at the API response in full, from the network tab of either our browser or the Playwright UI mode. The body looks as follows ```json book-api-response.json theme={null} { "id": 23, "title": "Achilles", "author": "Jayce Jomes", "genre": "novel", "price": "9.95", "rating": "★★★★☆", "stock": "1" } ``` What we’d like to do is fix that `"stock"` value to be the same every time. The process is easy. Before we load this page, let’s add a call to `page.route()` to modify the network requests made by the page: ```typescript mock-book-details.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('Book has correct details', async ({ page }) => { await page.route('*/**/api/books/23', async route => { const response = await route.fetch(); const json = await response.json(); json.stock="12" // Fulfill using the original response, while patching the response body // with the given JSON object. await route.fulfill({ response, json }); }); await page.goto('https://danube-webshop.herokuapp.com/books/23'); //Removed for brevity: checks of the book's title, genre, etc. await expect(page.locator('#app-content')).toContainText("Left in stock: 12"); }); ``` Now we can run this check as often as we like and be sure that the current stock will always be 12. Note that, if you’re using the Playwright UI, this mocking behavior will show up in the UI, as the API request will have two lines in the ‘Network’ tab Our e-commerce storefront

The ‘fulfilled’ listing will show the patched version of your response that went to the page. ## Going Further: Mocking Whole APIs If we’re able to intercept a request and change a single key/value pair, can we replace an entire payload from an API request? Sure, in fact we don’t need to wait for the real API to respond. the new version would look something like this: ```typescript full-mock-api.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('Book has correct details', async ({ page }) => { await page.route('*/**/api/books/23', async route => { const json = { "id": 22, "title": "Does the Sun Also Rise?", "author": "Ernst Doubtingway", "genre": "crime", "price": "9.95", "rating": "★★★★☆", "stock": "12" } await route.fulfill({json}); }); await page.goto('https://danube-webshop.herokuapp.com/books/23'); //Removed for brevity: checks of the book's title, genre, etc. await expect(page.locator('#app-content')).toContainText("Left in stock: 12"); }); ``` Now we no longer wait for this API response to come back and supply the whole response. As mentioned in the section above this has the benefit of delivering consistent results and removing the dependency on the API. In the Checkly UI, we can actually see that removing this API request and supplying a response significantly [speeds up](https://www.checklyhq.com/blog/speed-up-playwright-scripts-request-interception/) the overall test execution. Our e-commerce storefront

*Removing the API call makes this test take less than half the time* The distinct disadvantage is that we’re no longer checking that this API is available. If we’re [testing against production](https://www.checklyhq.com/guides/playwright-testing-to-monitoring/) it would be a good idea to still check that this API is returning data. ## Going Further: Replacing API responses to stress test our page The last use case for API mocking and response editing is simulating an unusual circumstance on a live page. For example, on our page we expect that a book with zero remaining stock should show up as sold out. Our e-commerce storefront

*Correct UI behavior when an item has zero stock* However, how can we test this behavior without setting up special ‘test’ conditions in the database? With our ability to patch API responses, we can force the stock response to be zero, and make sure the ‘add to cart’ button is *not* visible. ```typescript out-of-stock-test.spec.ts theme={null} import { test, expect } from '@playwright/test'; test('Book has correct details', async ({ page }) => { await page.route('*/**/api/books/23', async route => { const response = await route.fetch(); const json = await response.json(); json.stock="0" await route.fulfill({ response, json }); }); await page.goto('https://danube-webshop.herokuapp.com/books/23'); //Removed for brevity: checks of the book's title, genre, etc. await expect(page.getByRole('button', { name: 'Add to cart' })).not.toBeVisible(); await expect(page.locator('#app-content')).toContainText("Left in stock: 0"); }); ``` Sure enough, the behavior of our web store isn’t correct, with an out-of-stock item still showing as available to add to the cart. Our e-commerce storefront

*Whatever happens when you click ‘Add To Cart’ here it won’t be satisfying for the user* We can also stress test the UI in other ways: returning very large numbers for the data fields, or very long text blobs to make sure the UI still renders correctly. ### Editing API responses and visual regression tests As we find ourselves testing correct rendering of a page under unusual conditions, this does suggest a [visual regression test](/docs/detect/synthetic-monitoring/browser-checks/visual-regressions) to make sure a page looks correct. Visual regression tests shouldn’t be a default mode for functionality testing, but they are part of our toolkit and editing dynamic API responses can be an effective way of making sure that a page is always rendering with identical data for visual comparison. Our e-commerce storefront

*With fixed responses for dynamic page components, a visual regression test can create a golden file with a screenshot that isn’t expected to change* To go further, see our full documentation on [visual regression checks in Playwright and Checkly](/docs/detect/synthetic-monitoring/browser-checks/visual-regressions#visual-regression--snapshot-testing). ## Results and Conclusions: Playwright Mocking and Testing in Production With Playwright mocking you can remain tightly in control of API responses your page is using to render pages for users. This tool can help you in testing against production by ensuring that even with your production pages and data sources, your testing always sees consistent results. The result is more in-depth testing with higher reliability and fewer false alarms. And when a feature isn’t working correctly for users, you can find out before they complain. With Playwright's powerful API mocking capabilities, you gain precise control over the responses your web pages use to render content for users. This functionality is particularly valuable when testing against production environments, as it ensures consistent results even when dealing with dynamic data sources. By leveraging tools like `page.route()`, you can simulate various scenarios—such as out-of-stock items, unusual data inputs, or edge cases—without needing to alter your production database or backend systems. ### Key Takeaways: 1. **Consistency in Testing**: By mocking API responses, you can create stable and reliable tests that are not affected by fluctuating data, such as stock numbers or ratings. This reduces false positives and ensures your tests are always accurate. 2. **Flexibility**: Playwright allows you to modify specific parts of an API response or replace the entire payload, giving you the flexibility to test a wide range of scenarios, from stress testing to edge cases. 3. **Speed and Efficiency**: Mocking API responses can significantly speed up test execution by eliminating the need to wait for real API calls. This is especially useful for large-scale testing or when testing under time constraints. 4. **Improved Debugging**: By controlling API responses, you can isolate and test specific UI behaviors, making it easier to identify and fix issues in your application. 5. **Visual Regression Testing**: Mocking dynamic content ensures that visual regression tests are consistent, allowing you to create reliable "golden files" for comparison. ### When to Use API Mocking: * **Testing Dynamic Content**: Use mocking to stabilize tests for elements that frequently change, such as stock levels, prices, or ratings. * **Stress Testing**: Simulate unusual or extreme conditions (e.g., zero stock, large numbers, or long text) to ensure your UI handles them gracefully. * **Edge Case Testing**: Test how your application behaves under rare or unexpected conditions without needing to replicate those conditions in your production environment. * **Visual Regression Testing**: Ensure consistent screenshots for visual comparisons by fixing dynamic content in place. ### Limitations and Considerations: * **Real API Availability**: While mocking is powerful, it’s important to occasionally test against real API responses to ensure your production APIs are functioning correctly. * **Maintenance**: Mocked responses need to be updated if the structure of your API changes, so keep your mocks in sync with your API contracts. * **Over-Mocking**: Avoid mocking everything, as this can lead to tests that don’t reflect real-world behavior. Use mocking strategically to complement, not replace, real API testing. Playwright’s API mocking capabilities empower you to create robust, reliable, and efficient tests for dynamic web applications. By combining mocking with other testing strategies—such as visual regression testing and production monitoring—you can ensure your application delivers a consistent and high-quality user experience. Whether you’re testing edge cases, improving test speed, or ensuring UI consistency, Playwright provides the tools you need to build confidence in your application’s behavior.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Handling Multiple Tabs with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/multitab-flows Control multiple browser tabs in Playwright, opening new pages in a context and interacting with each independently within a single test. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright enables us to control multiple browser tabs, albeit in different ways. ## Opening tabs directly If we are looking to open brand new tabs with which to interact, the setup is rather straightforward. ```ts multitab-open.spec.ts theme={null} import { test } from '@playwright/test' test('open multiple tabs', async ({ browser }) => { const context = await browser.newContext() const pageOne = await context.newPage() const pageTwo = await context.newPage() await pageOne.goto('https://www.checklyhq.com/') await pageTwo.goto('https://playwright.dev/') await pageOne.screenshot({ path: 'screenshot-tab-one.png' }) await pageTwo.screenshot({ path: 'screenshot-tab-two.png' }) await browser.close() }) ``` ## Handling links that open a new tab Controlling tabs that are opened after a click on an element on the page can be trickier. Let's explore this through an example: * Navigating to `https://checklyhq.com/docs`. * Opening a new tab by clicking the link to the Checkly YouTube channel which opens in a new tab. By allowing us to wait for the creation of a child tab with `page.waitForEvent`, Playwright enables us to "catch" it following a click on an element with `target="_blank"`, and then seamlessly interact with any of the currently open tabs. ```ts multitab-flows.spec.ts theme={null} import { test } from '@playwright/test' test('open multiple tabs', async ({ browser }) => { const context = await browser.newContext() const page = await context.newPage() await page.goto('https://www.checklyhq.com/docs/') const [newPage] = await Promise.all([ context.waitForEvent('page'), await page.getByRole('link', { name: 'Checkly on Youtube' }).click() ]) await page.screenshot({ path: 'screenshot-tab-old.png' }) await newPage.screenshot({ path: 'screenshot-tab-new.png' }) await browser.close() }) ``` ## Further reading 1. Official documentation on [Playwright's multi-tab scenarios](https://playwright.dev/docs/pages)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Use Playwright to Navigate and Interact with Web Pages Source: https://www.checklyhq.com/docs/learn/playwright/navigation Learn how to navigate web pages and implement effective waiting mechanisms. A guide ideal for developers looking to refine their automation scripts. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Every script that we will write will almost certainly do three key things: 1. Navigating to some web page 2. Waiting for something 3. Possibly getting a timeout 😐 ## Navigating Initial navigation to any page can happen in multiple ways. * Whenever your code does a `page.goto()`, or a `page.click()` on a link, you explicitly trigger a navigation. * The webpage you are on can also trigger a navigation by executing `location.href= 'https://example.com'` or using the `history.pushState()` API. In the example below we trigger two navigations: 1. The initial load of the page. 2. A navigation to the shopping cart by clicking a link ```ts basic-browser-navigation.spec.ts theme={null} import { test } from '@playwright/test' test('basic navigation', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.click('#cart') }) ``` Run this example as follows: ```bash theme={null} npx playwright test basic-browser-navigation.spec.ts ``` ## Waiting Waiting for something to happen is a crucial part of any automation script. In most cases, this is handled automatically by Playwright. For example, when you click a button, Playwright will wait for that button to be clickable before it actually clicks it. In the example below, we type an email address into an input field on a login modal. Playwright's `fill` method comes with built-in waiting functionality. ```ts basic-browser-waiting.spec.ts theme={null} import { test } from '@playwright/test' test('basic built-in waiting', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Log in' }).click() await page.getByPlaceholder('Email').click() await page.getByPlaceholder('Email').fill('test@test.com') }) ``` Run this example as follows: ```bash theme={null} npx playwright test basic-browser-waiting.spec.ts ``` Playwright actions perform so-called [actionability checks](https://playwright.dev/docs/actionability#introduction) before interacting with DOM elements. If you call `click()` on a locator, Playwright will ensure that: * your locator resolves to exactly one element. * the matching element is visible. * the matching element is stable. * the matching element can receive events and is not covered or obscured by other elements. * the matching element is enabled. Playwright's auto-waiting and actionability checks allow you to focus on the end-to-end test flow without worrying when and if elements become visible. Playwright evaluates this for you. ## Timeouts The `page.waitForNavigation()` method — but also similar methods like `page.reload()` and `page.goBack()` — all take some options that determine "how" it should wait and what the timeout limits are. These options come in two flavors: **1. Hard timeout** The time in milliseconds passed as the `timeout` property e.g. `page.waitForNavigation({ timeout: 2000 })`. We do not recommend using this if you do not explicitly need to. **2a. DOM event based** These two options are directly related to the events your browser emits when it has reached a certain loading stage. * `load`: This is the `page.goto()` default and very strict: your whole page including all dependent resources, i.e. images, scripts, css etc. * `domcontentloaded`: less strict: when your HTML has loaded. **2b. Heuristic based** This option is based on the heuristic that if (almost) all network connections your browser has are no longer active, your page has probably finished loading. * `networkidle`: consider navigation to be finished when there are no more than 0 network connections for at least 500 ms. Relying on `networkidle` is discouraged because it slows down your end-to-end tests. It's recommended to rely on auto-waiting and application state to evaluate when a page is ready.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

## Further reading 1. [Waits and Timeouts](/docs/learn/playwright/waits-and-timeouts/) 2. [Playwright general navigation docs](https://playwright.dev/docs/navigations) 3. [Playwright auto waiting](/docs/learn/playwright/waits-and-timeouts) # Learn How to Test, Monitor, and Automate with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/overview Learn more about Playwright automation & monitoring with Checkly. Explore how to automate your web with a reliable, programmable monitoring workflow. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright serves as the foundation for Checkly's synthetic [*Monitoring as Code*](/docs/concepts/monitoring-as-code). To get the most from Checkly, we put together a series of guides and examples to help you get started with Playwright. ## Getting Started Get started with Microsoft's Playwright for automation and user simulation. Learn how to install Playwright and get started with your first script. ## Check our Playwright Tips YouTube channel [The Checkly YouTube channel](https://www.youtube.com/@ChecklyHQ) includes over a hundred videos covering synthetic monitoring, Monitoring as Code and Playwright automation. Check out our [Playwright Tips playlist](https://www.youtube.com/playlist?list=PLMZDRUOi3a8NtMq3PUS5iJc2pee38rurc) for bite-sized videos on Playwright best practices, tips and tricks. ## Diving Deeper Playwright is extremely powerful and serves a wide range of use cases. We have collected in-depth articles for important scenarios and challenging automation flows. Learn about guidelines to write quality automation scripts. Learn how to automate realistic user scenarios from signup to checkout. Learn how to handle authentication in Playwright and reuse credentials. Learn how you can use Playwright to measure page performance. ## Popular Posts Learn how to debug common errors and bugs in your Playwright scripts. Learn how to control HTTP requests and responses. Working with selectors is key for writing high quality scripts. # Measuring Page Performance Using Playwright - Best Practices Source: https://www.checklyhq.com/docs/learn/playwright/performance Measure web page performance with Playwright, understand why speed affects users and revenue, and track browser performance metrics over time. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. The need for fast and responsive applications has never been greater because of the move from [desktop to mobile](https://gs.statcounter.com/platform-market-share/desktop-mobile-tablet/worldwide/). Still, web applications have been increasing in [complexity and size](https://httparchive.org/reports/page-weight). It is clear why the topic of webpage performance is more popular today than it ever was. This article gives a practical introduction to the whys and hows of web performance without getting lost in the depth of this massive topic. ## Why performance matters The time it takes for a service to become usable influences a user's perception. Helpful features, great design and other prominent characteristics become irrelevant when an online service is slow, and users navigate away. You can build the best web application in the world, but be mindful that each user has limited time to invest in your service to solve their problems. Exceed that amount, and you risk losing them to a different, more performant solution. Especially for new users, a fast experience is essential because they haven't been given proof of the quality of your service yet. ### A competitive differentiator There is a brighter side to the topic: if low performance can sink an online platform, high performance can very well help it rise to the top. Speed and responsiveness can be a service differentiator, prompting users to choose it over the competition. Therefore an investment in this area will almost always pay off. Some notorious real-world examples from known businesses include: 1. Pinterest decreased user wait time, and [increased both traffic and conversions](https://medium.com/@Pinterest_Engineering/driving-user-growth-with-performance-improvements-cfc50dafadd7). 2. Zalando applied small load time improvements and found a direct correlation with [increased revenue per session](https://engineering.zalando.com/posts/2018/06/loading-time-matters.html). 3. The BBC discovered that every extra page load second led to 10% of [users leaving the page](https://www.creativebloq.com/features/how-the-bbc-builds-websites-that-scale). ## Measuring performance Given the importance of page performance, it is no coincidence that browsers expose a ton of insights into [performance metrics](https://web.dev/metrics/). Knowing how your application scores against these *across time* will provide the feedback you need to keep it performant for your users. Several approaches can be combined to gather the best insights: 1. *Real user monitoring* to understand what performance actual end-users of your service are experiencing. 2. *Synthetic monitoring* to proactively gather intel on service performance and find issues before users stumble into them. 3. *Performance testing* to avoid releasing performance regression to production in the first place. 4. *Regular audits* to get an overview of your page's performance and suggestions on how to improve it, e.g. with tools such as [Google Lighthouse](https://developers.google.com/web/tools/lighthouse). ### Performance metrics - Google's Web Vitals With Google pushing for a faster web, [the Web Vitals metrics](https://web.dev/vitals/) should be on your radar. Metrics such as Time to First Byte (TTFB), Total Blocking Time (TBT) or First Contentful Paint (FCP) are good user experience indicators and worth monitoring. Google recommends focusing on the three most important ones – Largest Contentful Paint (LCP), First Input Delay (FID) and Cumulative Layout Shift (CLS). **These three metrics are considered the Core Web Vitals** and give a good idea of a page’s loading behavior, interactivity, and visual stability. Not all of Google's Web Vitals are suitable for synthetic monitoring and performance testing. **First Input Delay** relies on user interactions, and it's best measured using real user monitoring. Use **Total Blocking Time** as an interactivity metric in a lab setting instead. ## Web Performance evaluation with headless tools As much as we should be striving to build performant applications, we should commit to monitoring and testing performance to enable continuous feedback and rapid intervention in case of degradation. Playwright provides a great toolkit to power synthetic monitoring and performance testing. 1. Access to the Web Performance APIs. 2. Whenever testing against Chromium, access to the Chrome DevTools Protocol for traffic inspection, network emulation and more. 3. Easy interoperability with performance libraries from the Node.js ecosystem. ### Web Performance APIs Modern browsers support many APIs to gather web performance metrics and web vitals. #### Navigation and Resource Timing API The Navigation Timing and the Resource Timing performance APIs are W3C specifications. The [MDN docs](https://developer.mozilla.org/en-US/docs/Web/Performance/Navigation_and_resource_timings) define the scope of both: > Navigation timings are metrics measuring a browser's document navigation events. Resource timings are detailed network timing measurements regarding the loading of an application's resources. Both provide the same read-only properties, but navigation timing measures the main document's timings whereas the resource timing provides the times for all the assets or resources called in by that main document and the resources' requested resources. [The Navigation Timing API](https://developer.mozilla.org/en-US/docs/Web/API/Navigation_timing_API) allows us to retrieve timestamps of key events in the page load timeline. A Navigation Timing entry includes metrics such as the navigation response time, the used protocol and document load time. ```ts basic-performance-navigation.spec.ts theme={null} import { test } from '@playwright/test' test('basic performance navigation', async ({ page }) => { await page.goto('https://danube-web.shop/') const navigationTimingJson = await page.evaluate(() => JSON.stringify(performance.getEntriesByType('navigation')) ) const navigationTiming = JSON.parse(navigationTimingJson) console.log(navigationTiming) }) ``` ```json theme={null} [{ "name": "https://danube-web.shop/", "entryType": "navigation", "startTime": 0, "duration": 1243.7999999998137, "initiatorType": "navigation", "nextHopProtocol": "http/1.1", "workerStart": 0, "redirectStart": 0, "redirectEnd": 0, "fetchStart": 0.10000000009313226, "domainLookupStart": 1.2000000001862645, "domainLookupEnd": 11.100000000093132, "connectStart": 11.100000000093132, "connectEnd": 336.8000000002794, "secureConnectionStart": 102.89999999990687, "requestStart": 336.89999999990687, "responseStart": 432.39999999990687, "responseEnd": 433.70000000018626, "transferSize": 971, "encodedBodySize": 671, "decodedBodySize": 671, "serverTiming": [], "workerTiming": [], "unloadEventStart": 0, "unloadEventEnd": 0, "domInteractive": 1128.8999999999069, "domContentLoadedEventStart": 1128.8999999999069, "domContentLoadedEventEnd": 1130.8999999999069, "domComplete": 1235.3999999999069, "loadEventStart": 1235.3999999999069, "loadEventEnd": 1235.3999999999069, "type": "navigate", "redirectCount": 0 }] ``` [The Resource Timing API](https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/Using_the_Resource_Timing_API) allows us to zoom in on single resources and get accurate information about how quickly they loaded. For example, we could specifically look at our website's logo: ```ts basic-performance-resource.spec.ts theme={null} import { test } from '@playwright/test' test('basic performance resource', async ({ page }) => { await page.goto('https://danube-web.shop/') const resourceTimingJson = await page.evaluate(() => JSON.stringify(window.performance.getEntriesByType('resource')) ) const resourceTiming = JSON.parse(resourceTimingJson) const logoResourceTiming = resourceTiming.find((element) => element.name.includes('.svg') ) console.log(logoResourceTiming) }) ``` ```json theme={null} { "name": "https://danube-web.shop/static/logo-horizontal.svg", "entryType": "resource", "startTime": 1149.1000000000931, "duration": 96.89999999990687, "initiatorType": "img", "nextHopProtocol": "http/1.1", "workerStart": 0, "redirectStart": 0, "redirectEnd": 0, "fetchStart": 1149.1000000000931, "domainLookupStart": 1149.1000000000931, "domainLookupEnd": 1149.1000000000931, "connectStart": 1149.1000000000931, "connectEnd": 1149.1000000000931, "secureConnectionStart": 1149.1000000000931, "requestStart": 1149.6000000000931, "responseStart": 1244.3000000002794, "responseEnd": 1246, "transferSize": 21049, "encodedBodySize": 20749, "decodedBodySize": 20749, "serverTiming": [], "workerTiming": [] } ``` #### Paint Timing API (`first-paint` and `first-contentful-paint`) [The Paint Timing API](https://developer.mozilla.org/en-US/docs/Web/API/PerformancePaintTiming) provides information on the first paint and the first contentful paint. Access the entries via `performance.getEntriesByType('paint')` or `performance.getEntriesByName('first-contentful-paint')`. ```ts basic-performance-paint-timing.spec.ts theme={null} import { test } from '@playwright/test' test('basic performance paint timing', async ({ page }) => { await page.goto('https://danube-web.shop/') const paintTimingJson = await page.evaluate(() => JSON.stringify(window.performance.getEntriesByType('paint')) ) const paintTiming = JSON.parse(paintTimingJson) console.log(paintTiming) }) ``` ````json theme={null} [ { name: 'first-paint', entryType: 'paint', startTime: 1149.5, duration: 0 }, { name: 'first-contentful-paint', entryType: 'paint', startTime: 1149.5, duration: 0 } ] #### Largest Contentful Paint API (`largest-contentful-paint`) [The Largest Contentful Paint API](https://developer.mozilla.org/en-US/docs/Web/API/Largest_Contentful_Paint_API) provides information on all large paints. Use this API to evaluate the Core Web Vital [Largest Contentful Paint](https://web.dev/lcp/) (LCP). > Large contentful paints are not a single event but rather event streams. A large paint can always be followed by an even larger one. > To evaluate the LCP initialize a `PerformanceObserver`, observe `largest-contentful-paint` entries and access the last emitted paint. ```ts largest-contentful-paint.spec.ts import { test } from '@playwright/test' test('basic performance largest contentful paint', async ({ page }) => { await page.goto('https://danube-web.shop/') const largestContentfulPaint = await page.evaluate(() => { return new Promise((resolve) => { new PerformanceObserver((l) => { const entries = l.getEntries() // the last entry is the largest contentful paint const largestPaintEntry = entries.at(-1) resolve(largestPaintEntry.startTime) }).observe({ type: 'largest-contentful-paint', buffered: true }) }) }) console.log(parseFloat(largestContentfulPaint)) // 1139.39 }) ```` #### Layout Instability API (`layout-shift`) [The Layout Instability API](https://developer.mozilla.org/en-US/docs/Web/API/Layout_Instability_API) provides information on all layout shifts. Use this API to evaluate the Core Web Vital [Cumulative Layout Shift](https://web.dev/cls/) (CLS). > Layout shifts are no single event but event streams. To calculate CLS initialize a `PerformanceObserver`, observe `layout-shift` entries and sum all shifts. ```ts basic-performance-layout-shift.spec.ts theme={null} import { test } from '@playwright/test' test('basic performance layout shift', async ({ page }) => { await page.goto('https://danube-web.shop/') const cummulativeLayoutShift: string = await page.evaluate(() => { return new Promise((resolve) => { let CLS = 0 new PerformanceObserver((l) => { const entries = l.getEntries() entries.forEach(entry => { if (!entry.hadRecentInput) { CLS += entry.value } }) resolve(CLS.toString()) }).observe({ type: 'layout-shift', buffered: true }) }) }, '0') console.log(parseFloat(cummulativeLayoutShift)) // 0.0001672498 }) ``` #### Long Task API (`longtask`) [The Long Task API](https://developer.mozilla.org/en-US/docs/Web/API/Long_Tasks_API) provides information about all JavaScript executions taking 50 milliseconds or more. Use this API to evaluate the Web Vital and lab metric [Total Blocking Time](https://web.dev/tbt/) (TBT). > Long Tasks are no single event but event streams. To calculate TBT initialize a `PerformanceObserver`, observe `longtasks` entries and sum the differences to the maximal JavaScript execution time of 50 milliseconds. ```ts basic-performance-long-task.spec.ts theme={null} import { test } from '@playwright/test' test('basic performance long task', async ({ page }) => { await page.goto('https://danube-web.shop/') const totalBlockingTime: number = await page.evaluate(() => { return new Promise((resolve) => { let totalBlockingTime = 0 new PerformanceObserver(function (list) { const perfEntries = list.getEntries() for (const perfEntry of perfEntries) { totalBlockingTime += perfEntry.duration - 50 } resolve(totalBlockingTime) }).observe({ type: 'longtask', buffered: true }) // Resolve promise if there haven't been long tasks setTimeout(() => resolve(totalBlockingTime), 5000) }) }, 0) console.log(parseFloat(totalBlockingTime.toString())) // 0 }) ``` ### Chrome DevTools for performance If the browser performance APIs are not enough, the Chrome DevTools Protocol offers many great performance tools for us to leverage with Playwright. One important example is network throttling, through which we can simulate the experience of users accessing our page with different network conditions. ```ts basic-performance-emulation.spec.ts theme={null} import { test } from '@playwright/test' test('basic performance emulation', async ({ page }) => { const client = await page.context().newCDPSession(page) await client.send('Network.enable') await client.send('Network.emulateNetworkConditions', { offline: false, downloadThroughput: (4 * 1024 * 1024) / 8, uploadThroughput: (3 * 1024 * 1024) / 8, latency: 20 }) await page.goto('https://danube-web.shop/') }) ``` The DevTools Protocol is quite extensive. We recommend exploring the [documentation](https://chromedevtools.github.io/devtools-protocol/) and getting a comprehensive overview of its capabilities. ### Additional performance libraries Lighthouse can easily be used programmatically with Playwright to gather values and scores for different metrics, like [Time To Interactive (TTI)](https://web.dev/interactive/): ```js theme={null} const chromeLauncher = require('chrome-launcher') const { chromium } = require('playwright') const lighthouse = require('lighthouse') const request = require('request') const util = require('util') ;(async () => { const chrome = await chromeLauncher.launch() const resp = await util.promisify(request)( `http://localhost:${chrome.port}/json/version` ) const { webSocketDebuggerUrl } = JSON.parse(resp.body) const browser = await chromium.connect({ wsEndpoint: webSocketDebuggerUrl }) const { lhr } = await lighthouse( 'https://danube-web.shop/', { port: chrome.port }, null ) console.log('Report complete for', lhr.finalUrl) console.log(` Time To Interactive - Score: ${lhr.audits.interactive.score}, Value: ${lhr.audits.interactive.numericValue} ${lhr.audits.interactive.numericUnit} `) await browser.close() await chrome.kill() })() ``` All above examples can be run as follows: ```sh theme={null} $ node measure-performance.js ``` ## Further reading 1. The comprehensive [MDN Web Performance documentation](https://developer.mozilla.org/en-US/docs/Web/Performance) 2. [Automatically finding accessibility issues with Playwright](https://www.checklyhq.com/blog/integrating-accessibility-checks-in-playwright-tes/) 3. [web.dev's performance section](https://web.dev/learn/#performance) 4. [Getting started with Chrome DevTools Protocol](https://github.com/aslushnikov/getting-started-with-cdp) by Andrey Lushnikov 5. [Get Started with Google Lighthouse](https://developers.google.com/web/tools/lighthouse#get-started)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Scrape Data Behind a Login with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/scraping-behind-login Scrape data that sits behind a login wall with Playwright by combining UI automation and scraping, illustrated by totalling Amazon order history. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright can be particularly useful when scraping data accessible only behind a login wall. This article shows a practical example of such a case. ## Scraping expenses on Amazon For our example, we will be logging in to our Amazon account and scraping the price off each order in the previous year, then adding them all up to show us our total Amazon expenditures over that period of time. A combination of UI automation and scraping will allow us to first log in to the platform, and then to retrieve the information about all our orders. ```js scraping-example-purchases.js theme={null} const { chromium } = require('playwright') ;(async () => { const currency = process.env.CURRENCY const amazonUrl = process.env.AMAZON_URL const amazonUser = process.env.AMAZON_USER const amazonPassword = process.env.AMAZON_PASSWORD const browser = await chromium.launch() const page = await browser.newPage() await page.goto(amazonUrl) await page.setViewportSize({ width: 1200, height: 1822 }) await page.click( '#nav-signin-tooltip > .nav-action-button > .nav-action-inner' ) await page.type('#ap_email', amazonUser) await page.click('#continue') await page.type('#ap_password', amazonPassword) await page.click('#signInSubmit') await page.click('#nav-link-accountList > .nav-long-width') await page.click( '.ya-card__whole-card-link > .a-box > .a-box-inner > .a-row > .a-column > div' ) await page.click('#a-autoid-1-announce > .a-dropdown-prompt') await page.click('#orderFilter_2') let filteredPrices = [] let prices = [] await page.waitForSelector('.a-normal') const pages = await page.$$('.a-normal') for (const singlePage of pages) { await singlePage.waitForSelector('.a-last') prices = await singlePage.$$eval( '.a-column:nth-child(2) .a-color-secondary.value', (nodes) => nodes.map((n) => n.innerText) ) filteredPrices = filteredPrices.concat( prices.filter((price) => price.includes(currency)) ) await singlePage.waitForSelector('li.a-last') await singlePage.click('li.a-last') } const cleanPrices = filteredPrices.map((x) => { return x.replace(',', '.').replace(/[^\d.,-]/g, '') }) const totalExpense = cleanPrices.reduce( (a, b) => parseFloat(a) + parseFloat(b), 0 ) console.log(`Total expense for the year: ${currency} ${totalExpense}`) await browser.close() })() ``` This example is only intended for learning purposes. Always make sure the website you are planning to scrape allows such behaviour. Run the above examples as follows, making sure to choose the right Amazon URL and currency: ```sh theme={null} CURRENCY=EUR AMAZON_URL=https://www.amazon.de AMAZON_USER= AMAZON_PASSWORD= node scraping-example-purchases.js ``` ```sh theme={null} SET CURRENCY=EUR SET AMAZON_URL=https://www.amazon.de SET AMAZON_USER= SET AMAZON_PASSWORD= node scraping-example-purchases.js ``` > Under the hood, Amazon can change quite quickly. You might need to adjust the locators and/or flow slightly to have the script work for you. > ⚠️ Websites might [restrict headless browser traffic](/docs/learn/playwright/challenging-flows/) in order to protect their users from fraud. 2FA will also interfere with the script if enabled. ## Takeaways 1. We can scrape information available behind a login wall with Playwright. 2. Some websites might not allow scraping. Always make sure you check their terms of service beforehand. ## Further reading 1. [Basic scraping](/docs/learn/playwright/web-scraping/) with Playwright

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Using Script Recorders with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/script-recorders Speed up script creation with Playwright recorders like codegen and the VS Code extension, then refine the generated selectors and waits for reliability. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. With Playwright, writing scripts manually is not the only option. Since browser automation became possible, there have always been tools trying to simplify script creation. Such instruments aim to help users with little or no scripting skills use automation tools, while also saving more expert users time when creating brand new scripts. Recorders can be used to quickly generate code for a scenario, saving time users would otherwise have to spend inspecting the various pages to find valid selectors. When creating multiple scripts, this adds up to a noticeable amount of time saved. ## How to record automation scripts with Playwright ### Record Playwright scripts with `codegen` Playwright provides multiple ways to record browser automation scripts. The recommended approach is to leverage [the built-in `codegen` command](https://playwright.dev/docs/codegen-intro#running-codegen). Playwright codegen example

> Pro tip: if you're using [the Playwright VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright#record-new-tests), you can also record and run scripts in your editor. ## Using recorders effectively Regardless of your chosen approach, you will want to inspect the output scripts to make sure they are doing what you need them to in the fastest and most reliable way possible. Double-check the newly created scripts and tweak it when necessary, especially keeping an eye out for: 1. Selectors, which should be in line with common [best practices](/docs/learn/playwright/selectors/). 2. [Waits](/docs/learn/playwright/navigation/), which should ensure the right element is present and/or ready for interaction at the right time; also, make sure you get rid of unnecessary waits. 3. Any sort of needless duplication. ## Takeaways 1. Recorders are a powerful tool to speed up script creation. 2. Recorded scripts should always be inspected and possibly tweaked to ensure correctness and efficiency. 3. Script maintenance is still required after scripts have been recorded. ## Further reading 1. [Playwright Codegen](https://playwright.dev/docs/codegen-intro#running-codegen) documentation.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Work with Selectors in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/selectors Learn to write solid scripts using Playwright. Discover how to choose stable selectors and start enhancing your skills today. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. When testing and monitoring websites end to end with Playwright, choosing the right locators is crucial. Proper locators help create tests that are less flaky and more reliable. Let's explore user-first locators and how to filter locators for more robust tests. ## Example Scenario Consider a simple web page with a headline, a button, and a status update box. Clicking the button updates the status and throws confetti. Here's how we can test this. a demo page

## Why we don't want to find a page element with CSS selectors If you're used to some older systems for automation and page manipulation like jQuery, we might be tempted to use CSS selectors. If we inspect this page we'll find, sure enough, that there is a class applied to this button that should select it in the page. examining the page with an inspector

We could click this button in a test with a line like `await page.locator("button.button-frontpage-style").click()` and that would work, but it's not recommended. What is the problem with this approach? Our users look at the page and want to find a button, they don't look for a CSS class, so our test is no longer emulating a user path. If a frontend engineer changes the class for style reasons to read `button-hero-panel-style` then our test will return a false positive: showing a problem where there is none. If our button text is coming from a CMS, and it breaks, the button text could change to "HEROBUTTON\_TXT" and our test would still pass, despite the UI being broken for the user, a false negative. Due to the reasons above, the Playwright project encourages you to not use CSS locators, and it's a good idea to follow the standards set down in the project! ### Instead of CSS selectors, try User-First Locators Playwright offers a number of locators that are based on page role, a more functional view of the page than finding by CSS or HTML. Use [the recommended built-in locators](https://playwright.dev/docs/locators#quick-guide) like `getByRole`, `getByText`, and `getByLabel`, which will all work the same in the test that they will for the user, even if the user is using an unusual browser build, mobile device, or even a screen reader! ## User-First Locators in Action Let's replace `page.locator` with `getByRole` to locate the button by its role and accessible name: ```ts theme={null} await page.getByRole('button', { name: 'click me' }).click() ``` Inspect elements using Chrome DevTools to find their accessibility roles and names. showing the element's role

This method ensures that tests remain stable even if class names or HTML structure change. As an added bonus, the name matching done by Playwright is case-insensitive, and matches by substring so you could change your selector to `{ name: 'click' }` and the test would pass, with no failures even if your marketing team relabels the button to read "Click me now!" This change means that if the button label changes to something like "HEROBUTTON\_TXT" the test will fail as expected. ### What to do when `getByRole` and `getByLabel` won't work on your page One of the reasons I support using user-first locators is what happens when you realize that these selectors don't work on your page. If role and label aren't set for many of your page items, or you find that all your page items have the same labels and roles, it indicates that your page has some serious accessibility issues. After all, if nothing has unique labels it means that screen readers and keyboard navigation tools won't work correctly on the page, meaning many users won't be able to navigate your pages. That should start a conversation among developers and QA people about improving the structure of your pages, which will fix testing and the experience for all users. In the short term, take a look at all of Playwright's locators, as there are definitely techniques that will find any element on your page, but I encourage you to use this as a jumping off point for tech debt conversations in the future. ### Handling Multiple Matching Elements If a locator matches multiple elements, Playwright's strict mode will fail. For example, using `getByRole('button')` might match several buttons. You'll get an error that reads something like: ``` Error: locator.click: Error: strict mode violation: getByRole('button', { name: 'click' }) resolved to 2 elements: ... ``` Note that in strict mode this test will fail even if the first result, or all the results, passes the rest of the test. Resolve this by specifying the position or filtering elements. #### Position-Based Selection When we first run into this error, the easiest solution is just to specify a position in a list. The simplest being to take the first result: ```ts theme={null} await page.getByRole('button').first().click() ``` I run into the error listed above so often, I get in the habit of adding first() to my locators almost every time I write a locator. 😅 If you want to be more specific about list position, use: ```ts theme={null} await page.getByRole('button').nth(3).click() ``` The .nth() function is zero-indexed, so .nth(3) will select the fourth item in the results. Positional selectors work, but they can be brittle, if we have dynamic page content, selecting anything but the first result may present a false positive. To be really sure that we're pointing to the correct element in a dynamic list, do a bit of element filtering. #### Element Filtering in Playwright If we imagine an e-store interface, with a number of results only some of which are available, we can see that a position selector isn't going to test reliably. too many matches

If we inspect the page we'll see that these are list items, so a selector that makes sure the item is available before checking for the button would read as follows: ```ts theme={null} const button = page.getByRole('listitem') .filter({ hasText: 'available' }) .getByRole('button', { name: 'buy' }) await button.click() ``` This test would now fail about when the user would expect it to: when they see a list of items, all marked as sold out. ## Best Practices for Page Locators There is no one-size-fits-all solution for locators. Here are some tips: * **Avoid Implementation Details**: Use locators that reflect user actions rather than HTML structure. * **Use Built-In Locators**: `getByRole`, `getByText`, and similar locators improve test stability and maintainability. * **Data Test IDs**: For complex scenarios, using data test IDs can simplify locators and improve test reliability. ## See user-first locators in action Putting all these pieces together, [check out this video for a demonstration of user-first locators.](https://www.youtube.com/watch?v=9RJMNU4eNEc) ### Conclusion Choosing the right locators is key to creating stable and maintainable Playwright tests. User-first locators reduce flakiness and encourage good development practices. And if you use Playwright for [end-to-end testing](https://www.checklyhq.com/guides/playwright-testing-to-monitoring/), consider using Checkly for monitoring your production site, we’ve got a great story for empowering your developers with [Monitoring as Code](/docs/guides/getting-started-with-monitoring-as-code).

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Self-documenting tests - add automatic Playwright steps with Typescript Decorators Source: https://www.checklyhq.com/docs/learn/playwright/steps-decorators Structure complex Playwright tests with a single @step TypeScript decorator, replacing repetitive test.step calls to make reports easier to scan. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. You can write Playwright end-to-end testing code using JavaScript or TypeScript. Which one should you choose? When I started writing my first automated browser tests, I went with JavaScript because I couldn't be bothered with the type wrangling. I just wanted to get something off the ground quickly. YOLO, right? Today, though, there are two reasons why **I last wrote a JavaScript-first Playwright test a very long time ago**. First, if you're worried about massive type headaches, complicated TypeScript generics, and countless red squiggly lines in your editor when you "just" want to end-to-end test your product features, here's a secret: [**Playwright doesn't type-check your code**](https://www.checklyhq.com/blog/playwright-type-checking-and-linting/). It understands and compiles TypeScript to JavaScript, but you can still write JavaScript or use `any` types. Playwright will not judge you on your TypeScript skills and will run your tests. More importantly, though, TypeScript is the better choice when planning to maintain a test suite in the long term because its benefits outweigh the added complexity. Auto-completion alone will make your test creation more manageable, and even though nobody likes to deal with type errors, when your test suite becomes a software project on its own, TypeScript warnings will help you discover problems in your code early. A few days ago, I found another TypeScript feature that allows me to **structure complex end-to-end tests with test steps while writing less code**. Doesn't this sound exciting? Let me show you how you can replace repetitive `test.step` calls with a single `@step` decorator. Let's go! ## The first problem: Playwright reports can be challenging to scan and understand. By default, all your test actions and assertions will be a long wall of instructions shown in UI mode or your test reports. That's not a big deal for twenty instructions, but if you're testing a complex UI flow, your test instruction count will quickly hit a hundred actions. Looking at a report with that many instructions isn't great. way too many lines on a top level playwright report

way too many lines on a top level playwright report

To solve this problem, you can group your actions in Playwright test steps. Here's [a snippet right from the Playwright docs](https://playwright.dev/docs/api/class-test#test-step). ```js theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { await test.step('Log in', async () => { // ... }); await test.step('Outer step', async () => { // ... // You can nest steps inside each other. await test.step('Inner step', async () => { // ... }); }); }) ``` You define a step, give it a name and wrap your existing Playwright code in an async callback function. A complex test case becomes very readable with a few added test steps. ```js theme={null} test('Customer is able to create account', async ({ page }) => { await test.step('Customer can create account', async () => { // create account instructions }); await test.step('Customer can log out', async () => { // log out instructions }); await test.step('Customer can log in', async () => { // log in instruction }); }) ``` Look at this beautiful and well-structured HTML test report. I'm a fan! much nicer html test report

You might now ask, "Does this work when you implement POMs ([Page Object Model](https://playwright.dev/docs/pom))?". It does, but wrapping every public class method in a test step isn't a great experience, which brings us to another problem. ## The second problem: wrapping all methods in a test step is annoying. Here's an example `PlaywrightPage` POM to test the search on the official Playwright docs. ```js theme={null} export class PlaywrightPage { readonly page: Page readonly searchBtn: Locator readonly searchInput: Locator constructor(page: Page) { this.name = name this.page = page this.searchBtn = page.getByLabel("Search") this.searchInput = page.getByPlaceholder("Search docs") } async goto() { await this.page.goto("https://playwright.dev") } async search() { // add a test step 👇 await test.step("Search", async () => { await this.searchBtn.click() await this.searchInput.fill("getting started") await this.page.getByRole("link", { name: "Writing tests" }).click() await this.page.getByRole("heading", { name: "Writing tests" }).click() }) } } ``` Wrapping one method in a test step (`search` in this case) isn't a big deal, but wrapping every public POM method will quickly feel like unnecessary busy work. ```js theme={null} export class YourPageObject { async methodOne() { await test.step("methodOne", async () => { /* ... */ }) } async methodTwo() { await test.step("methodTwo", async () => { /* ... */ }) } async methodThree() { await test.step("methodThree", async () => { /* ... */ }) } // There has to be a better way. 👆 } ``` Is there a better way to quickly add test steps? ## The solution: automagically wrap your POM methods with TypeScript decorators. Playwright doesn't include magic tricks to avoid this repetition, but we can use TypeScript tooling to make things easier. A TypeScript benefit I haven't mentioned yet is that TypeScript is a compiler that transforms your `.ts` files into JavaScript. In practice this means, that you can use modern JavaScript features in TypeScript and transform them to JavaScript code supported in older browsers or runtimes. One of these modern JavaScript features is decorators. JavaScript Decorators aren't supported anywhere but work if you use TypeScript. ### JavaScript Decorators — a proposal that's been in the making forever. [The JavaScript decorator spec proposal](https://github.com/tc39/proposal-decorators) saw the light of day eight years ago and reached ECMAScript proposal stage three. Proposals on stage three are considered "ready to implement". ```js theme={null} @defineElement("my-class") class C extends HTMLElement { @reactive accessor clicked = false; } ``` Unfortunately, no browser has bothered to implement the new language feature yet. However, this doesn't hinder the TypeScript team from shipping JavaScript decorators. But what are they? If you look at the proposal, you'll find that: > Decorators are functions called on classes, class elements, or other JavaScript syntax forms during definition. This definition is somewhat cryptic. Let me explain the feature in my own words. > With decorators, you can access, replace, or wrap class methods (like a method in your POM class) with other functions with a very condensed and developer-friendly syntax. Wrapping class methods is what we need to avoid all these `test.step` instructions. Let's find out how to define a decorator! ### How to replace `test.step` instructions with TypeScript decorators Let's look at our POM class again. ```js theme={null} class PlaywrightPage { constructor(page: Page) { /* ... */ } async search() { // we want to remove the `test.step`... await test.step("Search", async () => { // and "somehow" wrap these Playwright instructions... await this.searchBtn.click() await this.searchInput.fill("getting started") await this.page.getByRole("link", { name: "Writing tests" }).click() await this.page.getByRole("heading", { name: "Writing tests" }).click() }) } } ``` If you look at the `search` method, we want to remove the `test.step` from within the function body and somehow wrap the Playwright instructions with a test step. This situation is a perfect use case for decorators. First, we must apply a new decorator. ```js theme={null} export class PlaywrightPage { constructor(page: Page) { /* ... */ } // Use `@` to define a decorator @step() async search() { await this.searchBtn.click() await this.searchInput.fill("getting started") await this.page.getByRole("link", { name: "Writing tests" }).click() await this.page.getByRole("heading", { name: "Writing tests" }).click() } } ``` To decorate a class method with a Playwright step, put a `@step` line before the class method definition. Now TypeScript will complain… a complaint from a typescript linter in an IDE

a complaint from a typescript linter in an IDE

… because the decorator isn't defined yet. Decorators are normal JavaScript functions. Let's define a new `step` function. ```js theme={null} // 1. Define the `step` decorator function step(target: Function, context: ClassMethodDecoratorContext) { // 2. Return a method that will replace the original class method return function replacementMethod(...args: any) { // 3. Call the original method with the same arguments return await target.call(this, ...args) } } ``` Let's untangle this new function. When TypeScript discovers the `@step` syntax, it will try to call a `step` function. This function can be available in the current scope, or you can import it from somewhere in your codebase. When found, the `step` decorator function will be called with a function reference to our original method (`target`) and must return another function (`replacementMethod`). This returned function will replace the decorated method. Thanks to some low-level JavaScript, you can then call the original method (`target`) with the passed-in arguments (`args`). When you now run your tests and call a decorated class method, nothing has changed yet, but we're ready to apply some compilation magic. Let's extend the decorator! ```js theme={null} function step(target: Function, context: ClassMethodDecoratorContext) { return function replacementMethod(...args: any) { // 2. Use the surrounding context to define the step name const name = this.constructor.name + "." + (context.name as string) // 1. Wrap the target method with `test.step` return test.step(name, async () => { return await target.call(this, ...args) }) } } ``` First, you can wrap your `target.call` in a `test.step`. Then, you only need to find a way to define a step name; ideally, it would relate to your original method. You might have noticed it already; the decorator function will be called not only with the function reference but also with `context`. The replacement methods will also run in the same this context as the original method. With these two things, we can combine the POM class name (`this.constructor.name`) and the method name (`context.name`) to define a relatable step name. If we rerun our test, the `@step` decorator will do its magic and automatically wrap POM methods in a nice Playwright test step showing us the POM class and method. Beautiful! a well documented test

You can now go wild and replace all these `test.step` calls with `@step` decorators! ```js theme={null} export class YourPageObject { @step async methodOne() { /* ... */ } @step async methodTwo() { /* ... */ } @step async methodThree() { /* ... */ } } ``` But what if you want to give your test steps a human-readable name? ### How to pass custom step names to a decorator To pass a custom step name to your decorator, you must change how you decorate the class methods. Instead of "just applying" a decorator with `@step` you can also execute your decorators with `@step()`. This change allows you to hand in arguments like a possible step name. ```js theme={null} export class YourPage { @step("A great method") async methodOne() { /* ... */ } } ``` But you can't execute your decorator methods yet. Add another function level to your decorator to hand in arguments. ```js theme={null} // 1. Make `@step` executable to enable function arguments function step(stepName?: string) { // 2. Return the original decorator return function decorator( target: Function, context: ClassMethodDecoratorContext ) { return function replacementMethod(...args: any) { // 3. Use `stepName` when it's defined or // fall back to class name / method name const name = stepName || `${this.constructor.name + "." + (context.name as string)}` return test.step(name, async () => { return await target.call(this, ...args) }) } } } ``` Rename your original decorator and wrap it in another function that will return it. Then, you can rely on function scope and reuse the step name argument (`stepName`) to define a new step name in your replacement method. If `stepName` isn't defined, the code above falls back to the class name / method name combination. **You can now name your test steps with a single line!** Unfortunately, this approach has one downside. Once you make your step decorator function executable, you must execute it everywhere. `@step` must become `@step()`. ```js theme={null} export class PlaywrightPage { constructor(page: Page) { /* ... */ } @step('Perform a simple search and check heading') // "Performan a simple ..." async search() { /* ... */ } @step() // "PlaywrightPage.login" async login() { /* ... */ } } ``` In my opinion, adding `()` is a very reasonable trade-off for the ability of setting custom step names, though. ## Conclusion What do you think? Is implementing decorators for Playwright test steps worth the added complexity? For me, the answer is a firm "Heck yeah!". The decorator code is quite complicated JavaScript / TypeScript, but you'll rarely touch this code. It's a typical "set up once and forget about it" case, and I bet you'll fall in love with the easy-to-use decoratorar one-liners quickly. If you want to see this decorator implementation, find [a complete example implementation next to more Playwright example code on GitHub](https://github.com/checkly/playwright-examples/), and if you have any questions or comments, drop into the [Checkly community](https://www.checklyhq.com/slack/). We're a lovely bunch and happy to help!

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Playwright Screenshots - How to Take and Automate Screenshots Source: https://www.checklyhq.com/docs/learn/playwright/taking-screenshots Capture screenshots with Playwright's page.screenshot command, take full-page images, and automate visual comparisons to catch regressions. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Headless browsers are fully capable of taking screenshots, which is very useful in troubleshooting failures or faulty scripts. Using additional libraries and tools, it is also possible to automate visual comparisons. ## Generating and saving screenshots The `page.screenshot` command allows us to save one or more screenshots of the current page to a specified path. Without any additional options, the size of the screenshot depends on the viewport size: ```ts basic-screenshot.spec.ts theme={null} import { test } from '@playwright/test' test('take a screenshot', async ({ page }) => { await page.setViewportSize({ width: 1280, height: 800 }) await page.goto('https://danube-web.shop/') await page.screenshot({ path: 'my_screenshot.png' }) }) ``` ## Full page screenshots Adding the `fullPage: true` option allows for the capture of full page screenshots, overriding the `height` parameter specified for our viewport: ```js theme={null} await page.screenshot({ path: 'my_screenshot.png', fullPage: true }) ``` ## Clipped screenshots Having our screenshot limited to a smaller portion of the viewport is also possible. All we need to do is specify the coordinates of the top left corner of the screenshot, plus `width` and `height`. We then pass these options to: ```ts basic-screenshot-clipped.spec.ts theme={null} import { test } from '@playwright/test' test('take a clipped screenshot', async ({ page }) => { const options = { path: 'clipped_screenshot.png', fullPage: false, clip: { x: 5, y: 60, width: 240, height: 40 } } await page.setViewportSize({ width: 1280, height: 800 }) await page.goto('https://danube-web.shop/') await page.screenshot(options) }) ``` The above examples can be run as follows: ```sh theme={null} npx playwright test basic-screenshots.spec.ts ``` ## Visual regression testing Playwright can be used to take screenshots of a page and compare them with a reference image. This is useful for visual regression testing, where we can detect changes in the UI that may have been introduced by code changes. The `expect(page).toMatchSnapshot()` command is used to take a screenshot and compare it with a reference image. If the images are different, the test will fail. ```ts visual-regression.spec.ts theme={null} import { test, expect } from '@playwright/test' test('visual regression', async ({ page }) => { await page.goto('https://danube-web.shop/') const screenshot = await page.screenshot() expect(screenshot).toMatchSnapshot('danube-web-shop.png') }) ``` ## Further reading 1. Official documentation for taking screenshots with [Playwright](https://playwright.dev/docs/screenshots) 2. Blog post from baseweb.design on the whys and hows of [visual regression testing](https://baseweb.design/blog/visual-regression-testing/) 3. Blog post from Gideon Pyzer looking at different visual [regression testing tools](https://gideonpyzer.dev/blog/2018/06/25/visual-regression-testing/)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Reuse code with custom test fixtures in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/test-fixtures Share setup and logic across Playwright tests with custom fixtures, extending built-ins like page and context to keep your test base DRY and maintainable. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright's test fixtures are a powerful feature for sharing setup and logic across tests, promoting code reuse and maintainability. With custom fixtures, you can share code across multiple tests, resulting in a test base that is easier to expand and easier to maintain. ## Understanding Playwright Test Fixtures At their core, fixtures in Playwright are predefined components like `page`, `context`, `browser`, and `browserName` that you can use across different test cases. They encapsulate functionality into modular blocks, simplifying test code. When writing multiple tests for a single site or service, you're likely to end up performing the same tasks over and over, so we'd like to add fixtures to our Playwright tests that developers can re-use, rather than copying and pasting big blocks of code. This helps us follow the DRY principle: don't repeat yourself when writing code! ## Why we need custom fixtures In our web shop, we've got a login process that we want to simulate and test before doing further account actions. To open a dialog and log in, we might use some lines like the following: ```ts theme={null} import { test, expect } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('https://danube-webshop.herokuapp.com/'); await page.getByRole('button', { name: 'Log in' }).click(); await page.getByPlaceholder('Email').click(); await page.getByPlaceholder('Email').fill('robert@moresby.com'); await page.getByPlaceholder('Password').click(); await page.getByPlaceholder('Password').fill('@!#KLjwsWe4@'); await page.getByRole('button', { name: 'Sign In' }).click(); }); ``` This is a fine practice for a single test, but if we have dozens that all require login as a first step, we'll find ourself copying and pasting this code several times. This means a lot of copied code, such that if we ever tweak the login process, we'll have to update this copied code in dozens of places, not good! ## Playwright and Fixtures Playwright and its test runner Playwright Test, were designed with a broader testing scope and includes first-class support for fixtures. This design choice streamlines the setup and teardown process, making tests cleaner and reducing redundancy. ### Built-in Support Playwright's test runner provides a rich set of predefined fixtures, such as `browser`, `context`, `page`, and `request`, and allows for easy customization and extension of these fixtures. This built-in support enables more structured and maintainable tests, with less manual setup required. ```js theme={null} import { test, expect } from '@playwright/test'; // Configure the native `page` fixture with pre-defined state test.use({ storageState: 'authenticated-state.json' }); test('My authenticated test', async ({ page }) => { // Test actions using the authenticated page }); ``` The ability to extend and customize fixtures in Playwright not only reduces boilerplate but also enhances test isolation and reusability across your test suite. ## Create a Custom Test Fixture in Playwright Test Imagine testing a web application where multiple tests require logging in. Repeating login logic in every test is inefficient and violates the DRY (Don't Repeat Yourself) principle. Let's implement **a fixture to reuse and share code across Playwright test cases**. ### Step 1: Extend the Test Object First, create a new test file or use an existing one. Here, we're testing a web application with Playwright, accessing environment variables, and performing UI interactions. To create a custom fixture, extend the `test` object provided by Playwright: ```js fixture.js theme={null} import { test as baseTest } from '@playwright/test'; const test = baseTest.extend({ webApp: async ({ page }, use) => { // Your setup logic here // Logging into the web application, etc... // ... console.log('setup'); // After logging in, pass the `page` // object to the test cases // // Whatever you call `use` with // will be available in your test cases await use(page); // Teardown logic, if any console.log('teardown'); }, }); ``` By using custom fixtures, test cases can reuse code defined in the `test.extend` call. In this case, all tests using the extended `test` method can rely on the `webApp` fixture. ### Step 2: Use the Custom Fixture With the `webApp` fixture defined, you can use it in your tests. The fixture code will only run when a test "requests" to use it. For example, the custom `webApp` fixture will only be executed if a test case uses it. Replace `page` with `webApp` and use the fixture in your test cases: ```js theme={null} test('My webApp test', async ({ webApp }) => { // Your test code here, using the already logged in webApp fixture }); ``` The `webApp` fixture encapsulates the login logic, making it accessible to any test that requires it. ### Step 3: Debugging and Validation Debugging is crucial to ensure your fixture works as expected. Playwright's inspector is invaluable here, allowing you to step through test execution and verify the setup and teardown processes. ### Step 4: Sharing Across Tests To share your custom fixture across different test files, encapsulate it in a separate module: ```js setup.js theme={null} import { test as baseTest, expect } from '@playwright/test'; export const test = baseTest.extend({ /* Your fixture definition */ }); export { expect }; ``` Then, in your test files, import this extended `test` and use the custom fixtures: ```js tests/your-test.spec.js theme={null} import { test, expect } from './setup'; ``` This approach keeps your tests clean and focused, with shared logic abstracted into fixtures. ## Replace `beforeEach` / `afterEach` Hooks with Automatic Fixtures As explained in the previous section, fixtures enable the sharing and distributing of code across Playwright tests. By default, fixture code is only executed when a test case accesses a fixture. But there are scenarios in which you always want to run your fixture code, like global `beforeEach` and `afterEach` hooks. Let's implement **an automatic fixture to log the current time before and after every test case**. ### Define an Automatic Fixture If you want to run code before and after each test case, you can reach for the commonly known `beforeEach` and `afterEach` hooks. In the following example, we log the current time before and after every test to the console. ```js theme={null} test.beforeEach(async ({}) => { console.log(`Start time: ${new Date()}`); }); test('my test', async ({ page }) => { expect(page.url()).toBe('https://my.start.url/'); }); test.afterEach(async ({}) => { console.log(`End time: ${new Date()}`); }); ``` This approach works fine, but if you want to run the same code for **every test**, you must place these `beforeEach` and `afterEach` calls in every `spec` file. Additionally, `beforeEach` and `afterEach` don't share the same function scope. This behavior leads to unexpected errors in some scenarios. A more convenient way to implement global test hooks is relying on automatic fixtures. Like the `webApp` fixture defined in the previous section, you must extend the `test` object to use automatic fixtures. ```js base.js theme={null} import { test as baseTest } from '@playwright/test'; export const test = baseTest.extend({ // Define an automatic fixture // Note: `timelogger` is defined as any array to provide an options object timeLogger: [ async ({}, use) => { // Do something before every test // ... console.log(`Start time: ${new Date()}`); // Start the test case // // Note: `use` is called without any arguments because // automatic fixtures run regardless if they're requested or not await use(); // Do something after every test // ... console.log(`End time: ${new Date()}`); }, // Specify that this fixture should be an automatic fixture { auto: true }, ], }); export { expect } from "@playwright/test"; ``` The main difference between default and automatic fixtures is that automatic fixtures run regardless. In our example, the `timeLogger` fixture will be executed for every running test. Note that you'll want to update your other test spec files to import `test` from our `base.js`, they probably have something like: ```js theme={null} import { test, expect } from '@playwright/test'; ``` You'll want to update them to read something like: ```js theme={null} import { test } from './fixtures'; import { expect } from '@playwright/test'; ``` ## Conclusion Custom test fixtures in Playwright streamline test setup and teardown, fostering code reuse and reducing boilerplate. Encapsulating common functionality, such as user logins or global setups, into fixtures ensures your tests are cleaner, more maintainable, and easier to read. ### Playwright Fixtures Explained on YouTube To discover more practical fixture examples, check out our [YouTube playlist](https://www.youtube.com/watch?v=hegZS46J0rA\&list=PLMZDRUOi3a8N067UNvkxXEThKlTII_OJ-), which covers fixtures in even more depth.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Testing APIs with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/testing-apis Use Playwright's request fixture to test REST and GraphQL APIs directly, sending requests and asserting on responses without opening a browser. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. You probably know that Microsoft’s Playwright is a solid tool for end-to-end testing, enabling you to control headless browsers and check essential user flows. But did you know that you can also use Playwright for API testing? If you didn’t, then this guide is for you. In this post, we’ll explore how Playwright can be used to test a GraphQL API (but don’t worry if you’re using REST; Playwright can handle any HTTP-based API). Let's dive into an example using the countries.trevorblades.com GraphQL API. ## Our demo GraphQL API In this API, you might request a list of countries along with their codes and languages. You can even customize the response to include additional fields like the language code. This is a GraphQL api, which is supported in Playwright. GraphQL works by sending a POST request with a query in the request body. Let’s see how to write a Playwright test for this API. ## The Test Case To get started, let’s look at a basic test using Playwright. Below is an api.ts file that defines a test case doing little more than ensuring that our GraphQL API is responding with expected data. ```ts api.spec.ts theme={null} import { test, expect } from '@playwright/test' test('the GraphQL API works', async ({ request }) => { const response = await request.post('https://countries.trevorblades.com/', { data: { query: ` query { countries { code languages { code } } } `, // request the code for each country and language }, }); const body = await response.json() console.log(body) // This logs the API response expect(body.data.countries).toHaveLength(250) }) ``` In this test, we're using the `request` fixture instead of the usual `page` fixture. See a little documentation on [playwright.dev](https://playwright.dev/docs/test-fixtures) on the different fixture types available. Using `request` gives us a clean new API context with no special headers. We send a basic request, log the whole response, and then check the length of the response. ## Running the test When you run `npx playwright test`, you should see the data returned by the API printed in the terminal. If everything is working, the test should pass. When running tests locally you’ll probably want to separate your API tests, the simplest path being to save your API tests with a different naming convention and then filter your `test` command like `npx playwright test api.spec.ts`. ## Adding assertions Previously we were just checking that the response had the correct length. `expect(body.data.countries).toHaveLength(250)` To confirm that the assertion works, you can deliberately break the test by expecting 251 entries instead. This should trigger a failure. To go a bit deeper, let's compare the response to a stored JSON file: ```ts api.spec.ts theme={null} import { test, expect } from '@playwright/test' import countryData from './response.json' test('the GraphQL API works', async ({ request }) => { const response = await request.post('https://countries.trevorblades.com/', { data: { query: ` query { countries { code languages { code } } } `, }, }) const body = await response.json() expect(body).toEqual(countryData) }) ``` This version requires that you provide a JSON file with the expected response (that response.json file that's imported at the top). That's not ideal, instead, let's use playwright's snapshot tool so that we can take an initial snapshot of the JSON and compare later runs to that initial version. We'll replace the line: `expect(body).toEqual(countryData)` with `expect(JSON.stringify(allCountries)).toMatchSnapshot()` Now we'll need to run our test locally at least once with the -update-snapshots flag, and we'll save an initial version of the JSON to compare later: `npx playwright test -update-snapshots` The nice thing about this method is, not having to fool around with the file system, it means we can run this test from automated testing and [monitoring platforms like Checkly](/docs/detect/synthetic-monitoring/browser-checks/visual-regressions) without having to send example JSON or encode it into our test code. ## Filtering your API response While we could parse the JSON of the large response we got in our previous test, a more focused test would make a filtered request from the API, and just examine that response. ```ts api.spec.ts theme={null} test('the GraphQL API works for one country', async ({ request }) => { const germanyResponse = await request.post('https://countries.trevorblades.com/', { data: { query: ` query { countries(filter: { code: { eq: "DE" } }) { code } } `, }, }) const germany = await germanyResponse.json() expect(germany.data.countries).toHaveLength(1) expect(germany.data.countries[0].code).toBe('DE') }) ``` This test now validates a filtered response, all within the same Playwright test case. ## Continuous Monitoring with Playwright and Checkly By running multiple small API requests from within Playwright, you can verify the underlying APIs you need for your pages. If you want to monitor all your API endpoints continually, check out Checkly’s API monitors. Checkly API monitors let you run on a schedule with [dynamic notification settings and retries](/docs/detect/synthetic-monitoring/api-checks/response-limits), run from multiple geographies, and even [run setup and teardown scripts](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown) to add logic for your test, and clean up after your test. API Monitoring Dashboard

The dashboard for a Checkly API check, complete with response times, failures and retries, and checks run from multiple locations As a bonus, Checkly’s API checks are cheaper to run than browser checks. ## Conclusion Playwright isn’t just for browser testing; it’s also a powerful tool for API testing. By combining it with a monitoring service like Checkly, you can ensure that your APIs are reliable and catch issues before your users do. If you’d like to see how a Checkly works with Playwright to give you a total picture of the state of your services check out our pricing, or join our Slack to meet other engineers going further with Playwright.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Test Coupons and Discounts with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/testing-coupons Verify coupon and discount codes with Playwright by adding items to a cart and asserting the total price changes by the expected amount after applying the code. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Webshops and subscription-based services often offer discounts through coupon codes. Applying a valid coupon code during checkout might reduce the price of one, several, or all items in the shopping cart. ## Steps While discount coupons will be applied in different ways depending on the service or shop they are relevant to, in most cases: 1. Having selected one or more products will be a prerequisite for applying the coupon 2. Entering a valid coupon will result in visible feedback, i.e. a reduction of the previous product/cart price The following example, running against our [test site](https://danube-web.shop/), will add a variable number of items to the cart, then proceed to compare the total price before and after applying the coupon. The coupon is reducing the price of the whole shopping cart by 20%, therefore we will be asserting that the discounted price is reduced by the right amount. For this step we have chosen [Chai](https://www.chaijs.com/api/assert/), but any solid assertion library will do. ```js coupon.js theme={null} const assert = require('chai').assert const { chromium } = require('playwright') const productsNumber = process.env.PRODUCTS_NUMBER || 3 ;(async () => { const browser = await chromium.launch() const page = await browser.newPage() await page.goto('https://danube-web.shop/') for (let i = 1; i <= productsNumber; i++) { await page.click(`.preview:nth-child(${i}) > .preview-author`) await page.click('.detail-wrapper > .call-to-action') await page.click('#logo') } await page.click('#cart') await page.waitForSelector('#total-price') const price = await page.$eval('#total-price', (e) => e.innerText) await page.click('.cart > label') await page.click('#coupon') await page.type('#coupon', 'COUPON2020') await page.click('.cart > div > button') const expectedDiscountedPrice = price * 0.8 const discountedPrice = await page.$eval('#total-price', (e) => e.innerText) assert.equal(discountedPrice, expectedDiscountedPrice) await browser.close() })() ``` Run this example as follows: ```sh theme={null} PRODUCTS_NUMBER=3 node coupon.js ``` ```sh theme={null} SET PRODUCTS_NUMBER=3 node coupon.js ``` ## Takeaways 1. We can simply verify that coupons are accepted, or also check that they command the right discount. 2. Assertion libraries are useful when non-trivial [assertions](/docs/detect/synthetic-monitoring/api-checks/configuration) are required.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to Test File Uploads with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/testing-file-uploads Test file upload functionality with Playwright by logging in and attaching a file, using a profile-image upload as a worked example. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Most services allow users to upload files to their accounts. This can be a profile picture, a document, or any other type of file. Testing this functionality is crucial to ensure that users can upload files without any issues. In this example, we will show you how to test file uploads using Playwright. ## Steps Account properties to verify can run the gamut from simple text to connected third party services. In this example, we will focus on a popular case: changing a profile image by uploading one of our own. On our [test site](https://danube-web.shop/), such a test could look as follows: ```ts title="file-upload.ts" theme={null} // Example code for testing file uploads import { test, expect } from '@playwright/test'; test('upload profile image', async ({ page }) => { // Login to the website await page.goto('https://danube-web.shop/login'); await page.fill('#email', process.env.USER_EMAIL!); await page.fill('#password', process.env.USER_PASSWORD!); await page.click('#login-button'); // Navigate to profile page await page.goto('https://danube-web.shop/profile'); // Upload file await page.setInputFiles('input[type="file"]', process.env.FILE_PATH!); await page.click('#upload-button'); // Verify upload success await expect(page.locator('#upload-success')).toBeVisible(); await expect(page.locator('#upload-success')).toContainText('Upload successful'); }); ``` **macOS/Linux:** ```sh theme={null} USER_EMAIL=user@email.com USER_PASSWORD=supersecure1 FILE_PATH=file.jpg npx playwright test file-upload.ts ``` **Windows:** ```sh theme={null} SET USER_EMAIL=user@email.com SET USER_PASSWORD=supersecure1 SET FILE_PATH=file.jpg npx playwright test file-upload.ts ``` Here, we are simply checking for a message giving us feedback on the status of the upload. Depending on the website we are testing, it might be possible to also download the profile image afterwards to run a comparison locally for a more robust check. ## Takeaways 1. Use environment variables to inject secrets. 2. Use `setInputFiles` (Playwright) to upload the file. 3. If possible, download the file from the platform and compare it with the one that was just uploaded. ## Further reading 1. Official documentation from [Playwright](https://playwright.dev/docs/input#upload-files)

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# How to run Playwright tests in parallel Source: https://www.checklyhq.com/docs/learn/playwright/testing-in-parallel Run Playwright tests in parallel or in sequence, distinguishing stateless from stateful tests to avoid conflicts and speed up your suite. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Playwright offers robust capabilities for automating browser tests. However, a common question among developers revolves around the best practices for structuring Playwright projects, especially when tests involve significant environment changes, resource creation, or database updates. This post describes strategies for running Playwright tests in parallel, sequence, or both while optimizing your testing workflow for efficiency and reliability. ## Understanding Test Types: stateful vs stateless Before getting into the technicalities of test execution, you must categorize your tests into two main types: stateless and stateful. * **Stateless Tests** do not rely on a specific state of the environment and do not alter any global configuration. Since they are isolated in their operations, running them in parallel is generally safe and efficient. Stateless tests won't interfere with each other. * **Stateful Tests** modify the environment by creating or deleting resources, or changing global configuration. Such tests can easily conflict if run in parallel, as one test's actions might disrupt another's expected environment state. Tests that start and rely on a state created by a previous test are typically considered antipatterns. Tests should create and clean up their own state to avoid cluttering test environments and enable parallelization. ## Playwright's Default Test Execution Before you start wondering how to parallelize all your Playwright tests, be aware that you probably run some of your tests in parallel already. By default, **Playwright runs your test files in parallel, and all tests in a single file are run in order and with the same worker process**. Playwright default execution mode

This default execution model is a sane default. It safely runs tests in a single file in order while still trying to parallelize all the `spec` files. The execution model can be changed in multiple ways if you want to parallelize more tests or have specific requirements: * set `fullyParallel: true` in your `playwright.config` * change a test file's execution model using `test.describe.configure` * limit the numbers of `workers` in your `playwright.config` or via the command line Let's look at common execution scenarios and their required configuration. ## Running Tests Sequentially While parallel execution is efficient, specific scenarios require running tests sequentially. This is especially true for destructive and stateful tests, where ensuring a predictable and unaltered environment state is crucial for test accuracy. If you want to run all project tests sequentially, you can turn off Playwright's parallelism entirely. Limit the number of worker processes to one process, and all tests from all files will run after another. But remember that you usually want to parallelize as many tests as possible to keep the test execution times low. You can limit the workers via the command line: ```sh theme={null} npx playwright test --workers=1 ``` Or set the `workers` property to `1` in your `playwright.config`. ```ts playwright.config.ts theme={null} export default defineConfig({ workers: 1, }); ``` There are multiple options if you want to run only a subset of your tests sequentially in one worker. ```sh theme={null} # Run all test files in `./tests/sequential` sequentially npx playwright test --workers=1 ./tests/sequential # Run all tests with the test tag `@sequential` sequentially npx playwright test --workers=1 --grep @sequential # Run all tests in the `sequential` project sequentially npx playwright test --workers=1 --project=sequential ``` You can run a subset of test files, grep for specific test tags, or define a separate project to run tests sequentially. None of these approaches is better; you must decide what fits your project best. ## Running Tests in Parallel Thanks to Playwright's automatic encapsulation and easy-to-use worker functionality, the test runner naturally excels at executing tests in parallel, significantly reducing the time it takes to execute your entire test suite. By default, running multiple workers is limited to test files. Still, you can configure Playwright to parallelize running all your non-destructive and stateless tests, each running in its isolated environment. To explicitly configure all your tests to run in parallel, you can utilize the `fullyParallel` option in your Playwright configuration file or project settings. This option instructs Playwright to maximize parallelism by launching separate workers for each test file. Playwright will now try to run as many tests in parallel regardless of where they're defined. ```ts playwright.config.ts theme={null} export default defineConfig({ fullyParallel: true, }); ``` However, there are even more configuration options, and the test execution order in test files and groups can still be tweaked another way. ## Playwright's different execution modes While you can control test execution order and parallelism via the `workers` and `fullyParallel` option, Playwright also allows you to configure each test file or group with predefined execution modes. ### The `serial` mode (sequential) If you want to run all your project tests using `fullyParallel`, but one file's test should run sequentially, use the `serial` mode. ```ts serial.spec.ts theme={null} import { test, type Page } from '@playwright/test'; /** * Run tests in this file sequentially and stop after a failure. * * -> If `One` fails `Two` won't be run. */ test.describe.configure({ mode: 'serial' }); test('One', async () => { /* ... */ }); test('Two', async () => { /* ... */ }); ``` Note that by using the `serial` mode, a test failure will stop all the subsequent tests from running. ### The `default` mode (sequential) If you want to run all your tests using `fullyParallel`, but one file's test should run sequentially while ignoring all the test results, use the `default` mode. ```ts default.spec.ts theme={null} import { test, type Page } from '@playwright/test'; /** * Run tests in this file sequentially regardless of their results * * -> If `One` fails `Two` is still run. */ test.describe.configure({ mode: 'default' }); test('One', async () => { /* ... */ }); test('Two', async () => { /* ... */ }); ``` The `default` mode will run all defined tests and ignore if tests fail on the way. ### The `parallel` mode If you don't want to parallelize all your tests with `fullyParallel` but still have one file with stateless tests that you want to run in parallel, use the `parallel` mode. ```ts default.spec.ts theme={null} import { test, type Page } from '@playwright/test'; /** * Run tests in this file in parallel. */ test.describe.configure({ mode: 'parallel' }); test('One', async () => { /* ... */ }); test('Two', async () => { /* ... */ }); ``` ## Leveraging Playwright's Flexibility Playwright's design accommodates various testing strategies, allowing developers to tailor test execution to their project's needs. Whether running tests in parallel to save time or sequentially to ensure environment stability, Playwright provides the tools necessary to configure your testing environment effectively. ### Tips for Organizing Tests * Use directories, tags or projects to separate stateful and stateless tests and apply appropriate parallel or sequential execution modes. * Name your test files strategically if relying on alphabetical execution order for sequencing. * Consider leveraging Playwright's capabilities for test sharding and annotations for more complex test suite organization. ## Conclusion Choosing between parallel and sequential execution in Playwright tests hinges on understanding your tests' nature and their impact on the application environment. By categorizing tests into stateful and stateless, developers can apply the appropriate execution strategy, ensuring efficiency and reliability in their E2E testing processes. Over on [the Checkly YouTube channel](https://www.youtube.com/checklyhq), Stefan gets hands-on with Playwright test Parallelism and execution modes.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# The Layers of the Testing Pyramid Source: https://www.checklyhq.com/docs/learn/playwright/testing-pyramid This article gives a tour of the testing pyramid’s layers with examples, and shows you how each part is required for a complete picture of system performance. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. When you begin your journey in QA, SRE, Development, or Operations, you’ll ask about testing and hear ‘we test that on a different layer of the pyramid’ or ‘we test that during unit testing.’ As you grow, you’ll suggest an end-to-end test and hear “that’s not end-to-end testing, that’s integration testing.” It seems as if everyone else understands the layers of the testing pyramid, and which tests belong where. Really though, the separate are fairly abstract, without clean divisions either technically or conceptually. This article gives a tour of the testing pyramid’s layers with examples, and shows you how each part is required for a complete picture of system performance. ## How do we know our app is working? We know our app is working by testing, but this begs the question: who should test what, when? Let’s take the tiniest possible failure: trailing whitespace in user data. Should the React frontend refuse to submit form fields with extra whitespace, should the application automatically remove whitespace before sending database requests, or should the DB controller automatically remove whitespace? The question: “in what layer should these checks happen?” brings us to the testing pyramid. ## The Testing Pyramid in Full Originally conceived by Mike Cohn, and then popularized by Martin Fowler, the testing pyramid shows categories of tests, with the slowest on top and the fastest on the bottom.

The pyramid shape refers to the ideal distribution of tests: you should have dozens or hundreds of unit tests for every one end-to-end test. Some modifications visible on my version above: * integration tests and unit tests are only separated by a dotted line since in practice they may not be technically distinct, and may even be written and run by the same team * A second axis arrow indicates that on the bottom of the testing pyramid, the tests are more isolated, while on the top tests are naturally very integrated. Next let’s briefly describe the layers, along with how they work in a modern cloud native architecture. ## Unit Tests - with examples A unit test is a highly isolated test that only works with a tiny chunk of code, possibly the code in a single file, or even a single function. A unit test is by its nature extremely high performance, and can run as frequently as the developer saves her work (or even with every entered newline). In the following example, we have the unit test for a single function called `noWhiteSpace` ```ts theme={null} //noWhiteSpace.test.ts import { noWhiteSpace } from './noWhiteSpace'; describe('noWhiteSpace', () => { it('should remove all whitespace from each string in the array', () => { const input = [' hello ', 'world ', ' foo bar ']; const expected = ['hello', 'world', 'foobar']; expect(noWhiteSpace(input)).toEqual(expected); }); it('should handle strings with internal whitespace', () => { const input = ['a b c', 'd e f', 'g h i j']; const expected = ['abc', 'def', 'ghij']; expect(noWhiteSpace(input)).toEqual(expected)); }); it('should handle tabs and newlines', () => { const input = ['\ttext\n', 'with\twhitespace\r\n']; const expected = ['text', 'withwhitespace']; expect(noWhiteSpace(input)).toEqual(expected); }); it('should return empty strings for whitespace-only elements', () => { const input = [' ', '\t', '\n\r']; const expected = ['', '', '']; expect(noWhiteSpace(input)).toEqual(expected); }); it('should handle an empty array', () => { expect(noWhiteSpace([])).toEqual([]); }); it('should not modify strings without whitespace', () => { const input = ['hello', 'world', 'typescript']; expect(noWhiteSpace(input)).toEqual(input); }); it('should handle mixed whitespace characters', () => { const input = [' h\te\nl\rl\to\t']; const expected = ['hello']; expect(noWhiteSpace(input)).toEqual(expected); }); }); ``` Some observations about unit tests: * They can (as this one does) work as effective documentation of what the tested code is supposed to do * Unit tests could be written before the actual code, a process called Test-driven Development (TDD) Unit tests are intended to show that an individual block of code does what it says on the tin, and with their speed of execution should be running during the developer’s process of writing new code. Unit tests may be gathered into larger blocks and run later in the process, but since the higher layers take more than a few seconds to run, unit tests are the only ones that can run at the speed developers write code. the SDLC

*Where unit testing fits in the Software Development Life Cycle (SDLC)* In a cloud-native environment, unit tests are the one layer that’s largely unchanged: even if all our environments starting with Test are on the cloud, it’s likely that writing code and running code both happen on the developer’s laptop. For cloud-based IDE’s, the process of writing and running unit tests remains similar, with the cloud environment running unit tests with high frequency. What are the limitations of unit tests? In the example above we tested a very generic function to remove whitespace, but in the real world most of our functions will be called something like `userPreferencesUpdate` , and while our unit tests can test the code within that function, it can’t test the function working with real users. As such the unit test will feed in some fake user data and look at the function’s output, without reference to how real data stores, queues, routers, or other services might react. A unit test, then, exists in a universe of ‘stubs’ and ‘mocks’ which simulate the other services that make up your environment. To go one step further and either run or simulate larger chunks of your application, we go on to Integration tests. ## Integration tests - something in the middle Integrations test are supposed to show how well your code (or ideally your whole microservice) runs in something like your production environment. The layer includes two largely separate concepts: * Running your code in an environment where most of your application’s services are really running, and checking how it performs, looking for errors * Creating stubs which give fake input to your microservice, and mocks which mechanistically simulate the other services, and testing that your service runs as designed While a stub is as simple as a stored JSON with an identical request object that will always go in to your service at test time, a mock is a more fuzzy concept, but generally it appears to operate like another service without doing the actual work. Here’s an example of a mock: ```tsx theme={null} //userUpdateMock.ts interface UpdateUserRequest { userId: string; newName: string; } interface UpdateUserResponse { userId: string; updatedName: string; timestamp: string; status: string; } export function mockUpdateUserName(request: UpdateUserRequest): UpdateUserResponse { // Validate the request if (!request.userId || !request.newName) { throw new Error('Invalid request: userId and newName are required'); } if (typeof request.newName !== 'string' || request.newName.trim() === '') { throw new Error('Invalid request: newName must be a non-empty string'); } // Simulate processing delay const processingDelay = Math.floor(Math.random() * 200) + 100; // 100-300ms // Return the mock response return { userId: request.userId, updatedName: request.newName.trim(), timestamp: new Date().toISOString(), status: 'success', }; } ``` In this example, the mock almost looks like application code, it even validates the input slightly! But think about it: no request is made to any data store, and therefore there’s no check to see if this is even a valid userID, much less whether we’ve successfully updated their info. When we drew a dotted line between integration tests and unit tests, it was to indicate that it’s never 100% clear the distinction between a (simple, fast) unit test and a (slower, more comprehensive) integration test. It’s likely that integration tests will be run somewhere in between the moment-to-moment development process and the deployment of code to production. ### Integration testing’s more rigid cousin: contract testing In microservice architecture, ideally microservices would interact in tightly controlled ways. The schema for communication, responses, and the logic of those responses can be clearly defined as rigid contracts between all services. Contract testing conceives a system where services can always be tested by sending fixed requests, and receiving responses that conform to those contracts. There are three primary concerns with an over-reliance on contract testing: * In trying to define completely fixed contracts on service interactions, you’re limiting the growth of your system as the complexity outstrips the ability of contract documentation. * Contracts often expect consistent, readable information in data stores that doesn’t perfectly reflect reality. * The nature of failures during service interaction is rarely due to ‘services not upholding their contracts’ and as such contract testing fails to find failures before users do. Integration testing of some sort is a necessary part of the deployment process. ## End-to-end testing: testing like the users do The testing period requires, an addition to reflect testing in the real world. After all, once all the pre-deployment testing is over, the production code rolled out, and the application is updated, don’t we all go check out the site to make sure it looks ‘right’? Manual Testing atop the testing pyramid

*Many’s the night on an on-call team I’ve used my phone to check our production site when a user-reported error wasn’t showing up on any dashboard* When your default action is to go and click around on the site to really know that things are working, you’re showing that your end-to-end testing doesn’t have your full confidence. End-to-end tests are supposed to simulate the users interacting with your service. Passing an end-to-end test suite shows that all business critical paths through your application are working as expected. With testing tools like Playwright, there should be no user behavior that you can’t simulate. ```tsx theme={null} await page.locator('#item-to-be-dragged').dragTo(page.locator('#item-to-drop-at')); ``` *Not even tricky UI behavior like drag-and-drop requires manual programming in Playwright* At base, the advantage of end-to-end testing is that it answers the real question of testing: ‘will this work for users or not?’ the other phases of testing answer questions more akin to: ‘does this code function without throwing errors?’ or ‘is this service fulfilling its contract to other services?’ Those are important questions for earlier phases of the SDLC. But once our tests are going into the hands of users, we must simulate that real user behavior and record if everything works. ### From E2E Testing to E2E Monitoring If we agree on the importance of end-to-end testing, it begs the question if testing should stop after deploy time. Thousands of unforseen interactions can break a production service even after the code is tested post-deployment, and our users are certainly using the service all the time. Shouldn’t our testing also continue after the code is out? For end-to-end testing on a cadence, there’s Checkly, which uses the power of playwright to test your sites, services, and APIs around the clock. [Get started today](https://www.checklyhq.com/docs/)! ## What problem is the testing pyramid trying to solve? While it sadly causes many arguments about which tests belong where or which layer is most important, the testing pyramid is meant to acknowledge that each test type has a role to play, and the differences in performance is natural. ## Conclusions: Every Layer of Testing Matters We care about testing because ultimately there’s one layer of testing that no one wants to use: our users. If there’s a bug, they’ll find it, if there’s holes in a data flow, their data will fall into it. If there’s an edge case our users will find it. And every time a user is the first one to detect a problem, it hurts their trust in our service.

*The testing phase no one wants to use* The testing pyramid provides a structured approach to ensuring software quality by balancing different types of tests—each with its own strengths and trade-offs. Unit tests offer rapid feedback during development, verifying small pieces of logic in isolation. Integration tests bridge the gap, ensuring components work together as expected, whether through real service interactions or carefully designed mocks. Finally, end-to-end tests validate the system from the user’s perspective, confirming that critical workflows function correctly in a production-like environment. While the boundaries between these layers can sometimes blur, their collective purpose remains clear: to catch issues early, reduce risk, and build confidence before code reaches users. A well-balanced testing strategy leverages all three layers, recognizing that no single type of test can guarantee success on its own. By understanding and applying the principles of the testing pyramid, teams can optimize their testing efforts—maximizing coverage, efficiency, and reliability throughout the software development lifecycle. Ultimately, the goal is not to rigidly categorize tests but to ensure comprehensive validation at every stage. Whether you're writing unit tests for a utility function, mocking service interactions in integration tests, or automating user journeys with Playwright, each layer contributes to a more robust, resilient application. The testing pyramid isn’t just a model—it’s a reminder that quality is built step by step, from the smallest unit to the full user experience. ## Dive Deeper into the Testing Pyramid with Real World Applications In a recent Webinar, the Checkly team got to talk about how the Testing Pyramid affects testing strategies for real world applications like React apps. Watch the whole session here: # Automating User Signup with Playwright Source: https://www.checklyhq.com/docs/learn/playwright/user-signup-automation Automate user signup flows with Playwright by filling registration forms, submitting them, and confirming the account was created through a UI change. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Signups are key transactions in most web platforms, and therefore prime targets for automation. Oftentimes, registering an account is where we will find longer forms asking the user to answer a variety of questions. Luckily, Playwright is quick enough to blaze through these in seconds. ## Steps The flow will often match the following: 1. Navigate to the signup form. 2. Fill in all text fields, check all boxes etc. 3. Submit the form by clicking a button. We will likely want to also check that some change occurred in the UI to confirm that the registration worked. ```ts signup.spec.ts theme={null} import { test, expect } from '@playwright/test' test('Sign up', async ({ page }) => { await page.goto('https://danube-web.shop/') await page.getByRole('button', { name: 'Sign up' }).click() await page.getByPlaceholder('Name', { exact: true }).fill('John') await page.getByPlaceholder('Surname').fill('Doe') await page.getByPlaceholder('Email').fill('user@email.com') await page.getByPlaceholder('Password').fill('supersecure1') await page.getByPlaceholder('Company (optional)').fill('Test Inc.') await page.getByLabel('Myself').check() await page.getByLabel('I would like to receive').check() await page.getByLabel('I have read and accept the').check() await page.getByRole('button', { name: 'Register' }).click() await expect(page.getByText('Welcome back, user@email.com')).toBeVisible() }) ``` Run this example as follows: ```sh theme={null} USER_EMAIL=user@email.com USER_PASSWORD=supersecure1 npx playwright test signup.spec.ts ``` ```sh theme={null} SET USER_EMAIL=user@email.com SET USER_PASSWORD=supersecure1 npx playwright signup.spec.ts ``` The normal signup flow might include asking the user to confirm their email address right away by navigating to a URL included in an automated email. Reliably replicating the steps needed to achieve that is not trivial. A possible solution to the issue is having the system under test distinguish between test sessions and normal user sessions, and skip the verification step for test sessions. A way to achieve this would be to check for a specific user agent ID which could be set as part of our test: ```js theme={null} await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \ (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 TEST_ID/'); ``` ## Takeaways 1. Use environment variables to inject secrets. 2. You might need to go through additional steps in case email confirmation or similar is required.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Dealing with waits and timeouts in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/waits-and-timeouts Replace flaky hard-coded timeouts in Playwright with auto-waiting and web-first assertions to make your tests faster and more reliable. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. When you're trying to test your site's core user flows with Playwright and are facing an element not being loaded, many take the shortcut of waiting for a fixed amount of time by adding a hard-coded timeout. Hard timeouts are an anti-pattern, as they lower performance, increase the chances of a script breaking, and often introduce test flakiness. Let's explore the problem, how those issues arise and how Playwright enables you to test your site user-first. ## The problem of hard waits and tools without auto-waiting Hard waits and timeouts only do one thing: they instruct Playwright to wait for the specified time. This makes them dangerous: they are intuitive enough to be favored by beginners and inflexible enough to create serious issues. Let's explore some issues in practical terms. Imagine a situation in which your script uses **a tool without smart automatic waiting**: then you need to wait until an element appears on a page to interact with it. To click a button **you must ensure that this button is available in the current browser session**. As a quick solution, you might consider adding a hard timeout. This could look something like the following: ```js theme={null} // a tool without built-in auto-waiting await page.waitFor(1000) // hard wait for 1000ms await page.click('#button-login') ``` In such a situation, the following two scenarios can happen: 1. You end up waiting a shorter time than the element takes to load! playwright hard wait time too short

Your hard wait terminates and your click action is attempted too early. The script terminates with an error, possibly of the ["Element not found"](/docs/learn/playwright/error-element-not-found/) sort. 2. The element loads before your hard wait has expired. playwright hard wait time too long

While the element is correctly clicked once your wait expires and your script continues executing as planned, you are wasting time. You'll never click the element once it's available, but always rely on a magic number to interact with your UI. This wasted time quickly adds up when your test suite grows and you're running multiple test suites. If you've ever experienced waiting for a test suite taking an hour, you know how painful it can be. Every second counts in browser automation and end-to-end testing! **Hard waits and timeouts are always either too short or too long.** And in the worst-case scenario, the fluctating load times make the wait sometimes too long and sometimes too short, making our script fail randomly. You should avoid this situation at all cost because it will result in unpredictable, seemingly random failures, also known as test flakiness. Flakiness, a higher-than-acceptable and unpredictable failure rate, is a major problem. It is a source of noise, making the system state you are trying to test or monitor harder to understand. If your end-to-end suite includes a high rate of test flakiness, the stakeholders who routinely need to investigate failures will rapidly lose confidence in your automation setup. **Your tests and automation scripts must be stable and hard waits should never appear in your end-to-end testing and monitoring scripts**. ## Avoiding hard waits by relying on Playwright's auto-waiting actions and web-first assertions To avoid these issues, you should forget that hard waits exist and adopt tools like [Playwright, which provide auto-waiting mechanisms](https://playwright.dev/docs/actionability). Let's revisit the previous example and use Playwright's core functionality. ```js theme={null} // Playwright with built-in auto-waiting // `click()` waits for the element to be visible, stable, ... await page.getByRole('button', { name: 'Login' }).click() ``` You'll probably notice that the script above no longer includes a `wait` statement. Hard waits are unnecessary in Playwright scripts because when you call a Playwright action such as `click`, `fill` or `selectOption`, Playwright automatically waits until a set of actionability checks pass. ### Playwright's actionability steps To ensure your automation and testing actions behave correctly, Playwright enables you to forget about timings. Your job is to define browser actions and expected UI results; Playwright will figure out the rest. If you want to click an element, Playwright will only interact with it when the element is ready and actionable. The following checks evaluate actionability: * Does your defined locator resolve to *exactly one element*? * Is the resulting element *visible*? * Is the resulting element *stable*? (it's not moving, animating or transitioning) * Can the resulting element *receive events*? (it's not covered or obscured by other elements) * Is the resulting element *enabled*? (it doesn't have a `disabled` attribute) When you instruct Playwright to perform an action, it will constantly check if an element matches your locator and is ready to be used. And only if there is an actionable element will it perform your defined action. playwright smart wait

This auto-waiting approach has two main advantages: 1. Your Playwright scripts will be as quick as possible because Playwright will interact with elements whenever they're ready. 2. You can focus on defining UI actions and the expected results instead of worrying about network calls and timings. ### Playwright's web-first assertions Auto-waiting Playwright actions help to avoid using hard waits when interacting with elements. But what if you want to wait for an element to appear on the page or reach a specific state before proceeding with your script? Playwright's web-first assertions (`toBe*`) also provide auto-waiting capabilities. For example, to wait for an element to become visible, it's recommended to use Playwright's `toBeVisible` assertions. `toBeVisible`, `toBeEnabled`, `toBeChecked` and many more included assertions are asynchronous and wait for the elements to reach a certain state. ```ts theme={null} // wait for this button to be visible, stable, ... and click it await page.getByRole('button', { name: 'Login' }).click() // wait for this button to be disabled await expect(page.getByRole('button', {name: 'Login'})).toBeDisabled() // wait for this button to be gone await expect(page.getByRole('button', {name: 'Login'})).toBeHidden() ``` If you pair web-first assertions with Playwright's auto-waiting actions, your scripts will be expressive, human-readable, and, most importantly, not include any hard timeouts. It's important to understand that auto-waiting web-first assertions also have synchronous counterparts. ``` // check if this element is visible at this moment (true/false) await page.getByRole('button').isVisible() // wait for this element to become visible await expect(page.getByRole('button')).toBeVisible() ``` It's recommended to reach for web-first assertions whenever possible to avoid flakiness. If you're not using Playwright Test (web-first assertions are only available in `@playwright/test`) and you want to wait for an element to be visible, use `waitFor`. ```ts theme={null} const button = page.getByRole('button', { name: 'Login' }) // wait for this button to be visible await button.waitFor() ``` In any case, a good knowledge of [locators](/docs/learn/playwright/selectors/) is key to enable you to select precisely the element we need to wait for. ## Other waiting mechanisms Generally, **it's recommended to rely on Playwright's auto-waiting and built-in web-first assertions**, but if you must, here are some other waiting mechanisms. ### Waiting on navigations and network conditions in Playwright When you can't wait for an element to appear on a page and want to explicitly wait for the network, one of the other `waitFor` methods. #### `page.waitForLoadState` `waitForLoadState` waits until the required load state has been reached. It defaults to the page `load` event but can also be configured to wait for `domcontentloaded` or `networkidle` (discouraged). ```ts theme={null} // wait for a page `load` event await page.waitForLoadState() ``` #### `page.waitForURL` `waitForURL` waits until a navigation to the target URL. It also defaults to the page `load` event but can be configured to wait for `commit`, `domcontentloaded` or `networkidle` (discouraged). ```ts theme={null} // wait for the page `load` event of the `/login` URL await page.waitForURL('**/login') ``` #### `page.waitForRequest` / `page.waitForResponse` You can also wait until a request is sent or a response is received with `waitForRequest` and `waitForResponse`. These two methods are key for implementing [request and response interception](/docs/learn/playwright/intercept-requests/). ```ts theme={null} // wait for a request being made after clicking a button const loginRequestPromise = page.waitForRequest('/login') await page.getByRole('button', { name: 'Login' }).click() const loginRequest = await loginRequestPromise // wait for a response to come back after clicking a button const loginResponsePromise = page.waitForResponse('/login') await page.getByRole('button', { name: 'Login' }).click() const loginResponse = await loginResponsePromise ``` ### Waiting for page events With Playwright, you can also directly wait for `waitForEvent`. ```ts theme={null} // wait for a new window or popup to open after clicking a button const popupPromise = page.waitForEvent('popup') await page.getByRole('button', { name: 'Open new window' }).click() const popup = await popupPromise ``` ### Waiting for page functions And for more advanced cases, you can pass a function to be evaluated within the browser context via `waitForFunction`. ```ts theme={null} // wait for a specific state in the browser window after clicking a button const secretInternalState = page.waitForFunction( () => window.localstorage.secretThing === 'true' ); await page.getByRole('button', { name: 'Login' }).click() await secretInternalState ``` ## Takeaways 1. Never use hard waits or timeouts. 2. Use auto-waiting instead. 3. Combine auto-waiting actions with web-first assertions to test UI state instead of implementation details.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Playwright Web Scraping - How to Extract Data from Websites Source: https://www.checklyhq.com/docs/learn/playwright/web-scraping Extract data from web pages with Playwright, reading element attributes and properties for testing assertions or general-purpose scraping. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. We call the action of extracting data from web pages *web scraping*. Scraping is useful for a variety of use cases: 1. In testing and monitoring, asserting against the state of one or more elements on a page. 2. In general, gathering data for a variety of different purposes. You can use Playwright as a library to scrape data from web pages, without also using Playwright for testing. ## Scraping element attributes & properties Below is an example running against our [test site](https://danube-web.shop/), getting and printing out the `href` attribute of the first `a` element on the homepage. That just happens to be our logo, which links right back to our homepage, and therefore will have an `href` value equal to the URL we navigate to using `page.goto()`: ```js title="basic-get-href-value.js" theme={null} // Example code for getting href attribute const { chromium } = require('playwright'); (async () => { const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto('https://danube-web.shop/'); const href = await page.getAttribute('a', 'href'); console.log('First link href:', href); await browser.close(); })(); ``` As an alternative, it is also possible to retrieve an [ElementHandle](https://playwright.dev/docs/api/class-elementhandle) and then retrieve a property value from it. Following is an example printing the `href` value of the first `a` element of our homepage: ```js title="basic-get-href-handle.js" theme={null} // Example code for getting href using element handle const { chromium } = require('playwright'); (async () => { const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto('https://danube-web.shop/'); const element = await page.$('a'); const href = await element.getAttribute('href'); console.log('First link href:', href); await browser.close(); })(); ``` > The `innerText` property is often used in tests to assert that some element on the page contains the expected text. ## Scraping lists of elements Scraping element lists is just as easy. For example, let's grab the `innerText` of each product category shown on the homepage: ```js title="basic-get-text-values.js" theme={null} // Example code for getting text values from multiple elements const { chromium } = require('playwright'); (async () => { const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto('https://danube-web.shop/'); const categories = await page.$$eval('.category-link', elements => elements.map(element => element.innerText) ); console.log('Categories:', categories); await browser.close(); })(); ``` ## Scraping images Scraping images from a page is also possible. For example, we can easily get the logo of our test website and save it as a file: ```js title="basic-get-image.js" theme={null} // Example code for scraping and saving images const { chromium } = require('playwright'); const axios = require('axios'); const fs = require('fs'); (async () => { const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto('https://danube-web.shop/'); const imageSrc = await page.getAttribute('img[alt="logo"]', 'src'); console.log('Image source:', imageSrc); const response = await axios.get(imageSrc, { responseType: 'stream' }); const writer = fs.createWriteStream('logo.png'); response.data.pipe(writer); await browser.close(); })(); ``` We are using [axios](https://github.com/axios/axios) to make a `GET` request against the source URL of the image. The response body will contain the image itself, which can be written to a file using [fs](https://nodejs.org/api/fs.html). ## Generating JSON from scraping Once we start scraping more information, we might want to have it stored in a standard format for later use. Let's gather the title, author and price from each book that appears on the home page of our test site: books with titles ready for scraping

The code for that could look like this: ```js title="basic-get-data-json.js" theme={null} // Example code for scraping data and generating JSON const { chromium } = require('playwright'); const fs = require('fs'); (async () => { const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto('https://danube-web.shop/'); const books = await page.$$eval('.book-item', elements => elements.map(element => ({ title: element.querySelector('.book-title').innerText, author: element.querySelector('.book-author').innerText, price: element.querySelector('.book-price').innerText })) ); fs.writeFileSync('books.json', JSON.stringify(books, null, 2)); console.log('Books data saved to books.json'); await browser.close(); })(); ``` The resulting `books.json` file will look like the following: ```json theme={null} [ { "title": "Haben oder haben", "author": "Fric Eromm", "price": "$9.95" }, { "title": "Parry Hotter", "author": "J/K Rowlin'", "price": "$9.95" }, { "title": "Laughterhouse-Five", "author": "Truk Tugennov", "price": "$9.95" }, { "title": "To Mock a Killingbird", "author": "Larper Hee", "price": "$9.95" }, ... ] ``` All the above examples can be run as follows: ```sh theme={null} $ node scraping.js ``` ## Further reading 1. [Playwright](https://playwright.dev/docs/test-assertions)'s official API reference on the topic 2. An [E2E example test](/docs/learn/playwright/testing-coupons/) asserting against an element's `innerText`

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Beginner’s Guide to Playwright Automation Source: https://www.checklyhq.com/docs/learn/playwright/what-is-playwright Learn how Playwright, a browser automation tool, simplifies testing across Chrome, Firefox, and WebKit. Start building more reliable web applications today. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. ## What is Playwright? Playwright is an open-source framework for cross-browser automation and end-to-end web application testing. It was designed to be a fast, reliable, robust, and evergreen test automation framework, and its API supports modern rendering engines that include Chromium, WebKit, and Firefox. Playwright tests run on Windows, Linux, and macOS, locally or on your continuous integration pipeline, and headless or headed. Playwright supports several programming languages, including TypeScript, JavaScript, Python, .NET, and Java. ## End to end testing with Playwright Playwright was created by the same engineering team that built Puppeteer. The Playwright team created Playwright to extend Puppeteer’s UI automation capabilities to all rendering engines, no longer just supporting Chromium. Microsoft engineers built on top of Puppeteer’s core features to provide a testing framework that met the challenges presented by today’s increasingly complex, feature-rich web applications and software development life cycles. Modern web applications are built as distributed systems that comprise separate components and services. This architecture is equally reflected in team organization with larger teams split into smaller pods, each responsible for a specific product requirement. Within this distributed and service-oriented model, automated end-to-end tests must simulate real-world user test scenarios and verify that an application’s moving parts work together, as intended. End-to-end tests provide disparate product teams with the confidence needed to determine if a web application is working or not. End-to-end tests have the reputation of being flaky and slow, while also requiring a lot of maintenance. These limitations are reflected in the Test Pyramid’s recommendation that end-to-end tests should be kept at a minimum due to their unreliability and difficulty in handling complex, feature-heavy, and service-oriented web applications. Playwright overcomes flaky and slow tests by communicating with rendering engines via the WebSocket protocol, instead of through multiple HTTP requests, as is done with other web automation testing tools like Selenium. Playwright communicates all requests through a single web socket connection that remains active until all tests have been completed. This reduces points of test failure during test execution, providing a stable and fast solution. ## Playwright Features While taking a simplified approach to its design, Playwright offers many features that make it unique, but competitive with several other leading test automation tools, like Cypress and Puppeteer. While we’ve discussed some of the key features in more detail above, some additional noteworthy features include: 1. **Mobile Emulation**: Playwright allows you to emulate mobile devices, such as iPhones and iPads, and test how your web application behaves on different screen sizes and orientations. 2. **Headless Mode**: You can run your tests in a headless browser, which is useful for CI/CD pipelines and for running tests in environments where a graphical interface is not available. 3. **Video Recording**: Playwright can record videos of your tests, which can be useful for debugging. 4. **Tracing**: Playwright can capture your application state at every step of your tests to provide detailed insights when debugging. 5. **Cross Browser Testing**: Playwright supports multiple browsers, including Chromium, Firefox, and WebKit, allowing you to test your application across different browsers. 6. **Programming Language Support**: Playwright is available in several programming languages, including JavaScript, TypeScript, Python, .NET, and Java, making it accessible to a wide range of developers. 7. **Comprehensive API**: Playwright provides a comprehensive API for automating browser actions, such as navigation, page interaction, and element manipulation. 8. **CI/CD Integration**: Playwright integrates seamlessly with popular CI/CD platforms, such as GitHub Actions, Jenkins, and CircleCI, allowing you to run your tests as part of your continuous integration and delivery process. 9. **Visual Testing**: Playwright supports visual testing, allowing you to compare screenshots of your web application and detect visual changes. 10. **Code Generation**: Playwright includes a code generator that can generate code for your tests, saving you time and reducing the risk of errors. ## Main use cases 1. **Test automation in modern web applications:** verifying that the features we are exposing our users/customers to are actually behaving as expected. 2. **Cross-browser testing:** ensuring applications are working consistently across browsers and rendering engines. 3. **Taking screenshots of web pages:** useful for a variety of different uses going from simple archiving to automated comparison for e.g. visual testing. 4. **Scraping web sites for data:** extracting data from websites for later retrieval or analysis. 5. **Automating interaction of web pages:** speed up and scale any sort of sequence of actions we would like to perform on a website automatically. ## Growth and Adoption Since its release in January 2020 by Microsoft, Playwright has experienced a surge in usage and popularity. As of January 2026, its GitHub repository has received 80k stars and 5k forks, and is currently averaging more than 32 million NPM downloads per week. Playwright is used by enterprise and open-source development projects alike, including Adobe Spectrum, Visual Studio Code, and React Navigation. ## Playwright as part of Checkly At Checkly, we believe that end-to-end testing is only one part of shipping reliable software. That's why we enable you to use your Playwright codebase not only for testing, but also for monitoring and automating your web applications. With Checkly, you can run your Playwright scripts in the cloud, on a schedule, and get alerted when something goes wrong. This way, you can ensure that your web applications are always up and running, and that your users have a stellar experience.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

## Further reading 1. [Official Playwright API documentation](https://playwright.dev/) 2. [Official Playwright GitHub repo](https://github.com/microsoft/playwright) # Best Practices for Writing Tests in Playwright Source: https://www.checklyhq.com/docs/learn/playwright/writing-tests Write valuable, maintainable Playwright tests by keeping scripts short, simple, and readable, so they keep telling the truth about your system as it evolves. If you're using Playwright for end-to-end testing, you should check out [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) and start testing in production. Headless browsers can be leveraged for testing in a variety of ways, and different scenarios do command different approaches. There are some general pointers most should follow in order to keep their tests *valuable*. Here, we define *valuable* as *sustainably expressing meaningful, truthful information about the state of a system*. A test that does not reliably fulfill these criteria should be fixed, if possible, or simply removed. ## Keep tests short If they run against a product with a UI that is evolving over time, scripts will need to be regularly updated. This brings up two important points: 1. Most scripts are not write-and-forget, so each script we write is one more script we will have to maintain. 2. Like all cases where code and refactoring are involved, *how* we write scripts can have a significant influence on how long this maintenance effort takes. Taking example from good software engineering practices, our scripts should strive for *simplicity, conciseness and readability*: 1. **Simplicity:** keep in mind the goal of the script, and keep away from overly complex solutions whenever possible. 2. **Conciseness:** simply put, do not be overly verbose and keep scripts as short as they can be. 3. **Readability:** follow general [best practices](https://blog.pragmaticengineer.com/readable-code/) around writing code that is easy to read. The faster we can read and understand a script we (or a teammate) wrote in the past, the quicker we can interpret its results and get to work on updating it and making it relevant again. Do not underestimate the compounding effect of having messy scripts across a large test base. ## Keep tests focused Automated tests are effective if they: 1. Correctly verify the status of the target functionality. 2. Return within a reasonable amount of time. 3. Produce a result that can be easily interpreted by humans. The last point is often overlooked. Scripts by themselves have no meaning if their results mean nothing to whoever is looking at them. Ideally, we want the opposite: interpreting a test success or failure should be close to instantaneous and give us a clear understanding of what is working and what is not. Oftentimes this is impeded by the tendency to have tests do too much. We can draw an example from two scenarios running against our [test site](https://danube-web.shop/): [E2E Checkout](/docs/learn/playwright/checkout-testing-guide/) and [E2E Coupon](/docs/learn/playwright/testing-coupons/). While these two have part of their flow in common, and we might be tempted to combine them to avoid a certain degree of duplication, merging them into a single test would obfuscate the meaning of a test failure as we would be testing two different features. If that combined test were to run red, we would be unable to tell whether the entire checkout is not working as expected, or whether users are just unable to redeem coupon codes. Unless we were to devote additional time to diving deep into the failure - which is exactly what we are trying to avoid. We can avoid this pitfall by making sure our tests are verifying only one feature each. Always check the assertions in your test: if they are spanning more than one feature, you would likely be better off splitting your test into multiple different ones. ## Keep tests independent In an effort to remove duplication, tests are often made dependent on the previous execution of one or more other tests. An example could be the following sequence of tests: ``` test1: - creates user1 - asserts user1 has been created test2: - logs in with user1 - asserts user1 has logged in successfully test3: - goes through checkout as user1 - asserts that checkout for user1 was successful ``` In this case, the success of each subsequent test depends on previous tests as much as on its own assertions. In other words, previous tests are driving a test's success criteria. This form of coupling can create headaches for us in the shape of: 1. **Lower test result readability:** we might need to backtrack and look into previous tests when trying to understand what a subsequent one is doing 2. **Harder maintenance:** changes might need to be applied across different scripts 3. **Lack of flexibility and parallelisation:** tests need to run sequentially in a specific order to work Having each test encapsulate all it needs to give us a meaningful answer is fundamental to avoid the above issues. In case of our example, a solution could look like: ``` test1: - creates user1 - asserts user1 has been created test2: - (setup: creates user1) - logs in with user1 - asserts user1 has logged in successfully test3: - (setup: creates user1) - (setup: logs in with user1) - goes through checkout as user1 - asserts that checkout for user1 was successful ``` Where: 1. Tests can be run in any order, even at the same time. 2. In the interest of code quality, we would be factoring out setup phases in order to easily reuse them across tests. [Jest](https://jestjs.io/docs/en/setup-teardown)'s `beforeEach` and `beforeAll` hooks are examples of useful helper functions that can help us achieve that. If the system you are testing allows it, provision/deprovision your users through APIs in your setup and teardown phases to save time and increase test reliability. ## Takeaways 1. Tests should be reliable and informative in order to be useful. 2. Keep tests short and focused on testing one feature. 3. Keep tests independent to maximise their parallelisation potential and reduce total runtime. ## Further reading 1. [Gergely Orosz](https://blog.pragmaticengineer.com/readable-code/) on writing readable code.

Bugs don't stop at CI/CD. Why would Playwright? Playwright logo

# Allowlisting Traffic Source: https://www.checklyhq.com/docs/platform/allowlisting-traffic Allowlisting traffic to Checkly There are cases in which you might have to allowlist Checkly traffic in your firewall, load balancer or other to prevent it from being blocked or skewing analytics data. Below are some solutions to help you achieve that. ## IP range allowlisting All of Checkly's monitoring traffic comes from a fixed source of IP addresses. Allowlisting Checkly traffic by IP address is possible by configuring your network to only allow traffic from these IPs. Fetch the current list of Checkly IP addresses: 1. A JSON array of all IPv4s currently used ```bash Terminal theme={null} curl https://api.checklyhq.com/v1/static-ips ``` 2. A JSON object with regions as keys and arrays of all IPv4s of that region as the value ```bash Terminal theme={null} curl https://api.checklyhq.com/v1/static-ips-by-region ``` 3. A simple txt file. One IPv4 per line. ```bash Terminal theme={null} curl https://api.checklyhq.com/v1/static-ips.txt ``` 4. A JSON array of all IPv6s currently used ```bash Terminal theme={null} curl https://api.checklyhq.com/v1/static-ipv6s ``` 5. A JSON object with regions as keys and arrays of all IPv6s of that region as the value ```bash Terminal theme={null} curl https://api.checklyhq.com/v1/static-ipv6s-by-region ``` 6. A simple txt file. One IPv6 per line. ```bash Terminal theme={null} curl https://api.checklyhq.com/v1/static-ipv6s.txt ``` We update this list very infrequently (6 months or less) so querying it with at reasonable frequency is recommended. ## Header/user agent allowlisting If you are using Cloudflare or a similar provider, one or more of your automated checks might trigger a [bot detection mechanism](https://www.cloudflare.com/learning/bots/what-is-bot-traffic/). If you want to prevent that from happening without exposing your website to any and all automated traffic, you might want to set up a new [firewall rule](https://developers.cloudflare.com/firewall/cf-firewall-rules/) allowing traffic from Checkly as long as it contains a specific header or sets a certain user agent. You can make the header and/or user agent specific to your own Checkly user account by grabbing the first eight digits of your unique user ID (or another fixed value of your choice), which you can find below your account name on the [Account Settings page](https://app.checklyhq.com/settings/account/). Embedding this value in your checks will enable them to be allowed through by your firewall rules. ### Allowlisting API checks using headers To allowlist API checks, allow traffic that contains a cookie in the shape of `Cookie: "checkly-account:"`, with `` being your shortened Checkly ID or other chosen value. You can then [set the Cookie header](/docs/detect/synthetic-monitoring/api-checks/configuration#headers) while editing your check. > You can set the header at group-level using [API check defaults](/docs/platform/groups#headers--query-parameters) to have it applied to every API check in the group. ### Allowlisting browser checks with user agents To allowlist browser checks, allow traffic with user agent containing `Checkly/`, with `` being your shortened Checkly ID or another chosen value. You will then be able to set up the matching user agent in your browser checks using Playwright's [`userAgent`](https://playwright.dev/docs/emulation#user-agent) property. ```ts user-agent.spec.ts theme={null} import { test } from '@playwright/test' const myUserAgent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 Checkly/abcd1234' test.use({ userAgent: myUserAgent }) test('my user agent test', async ({ page }) => { // ... }) ``` ```js UserAgent.spec.js theme={null} const { test } = require('@playwright/test') const myUserAgent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 Checkly/abcd1234' test.use({ userAgent: myUserAgent }) test('my user agent test', async ({ page }) => { // ... }) ``` ### Allowlisting Playwright check suites with user agents You can configure the user agent globally in your `playwright.config.ts` file to ensure all your Playwright tests use the Checkly user agent. **For all projects:** ```ts playwright.config.ts theme={null} import { defineConfig } from '@playwright/test' export default defineConfig({ use: { userAgent: "(Checkly, https://www.checklyhq.com)", }, }) ``` **For individual projects:** To preserve the browser's user agent while adding the Checkly identifier, append it to the device's user agent: ```ts playwright.config.ts theme={null} import { defineConfig, devices } from '@playwright/test' export default defineConfig({ projects: [ { name: 'chromium-checkly-agent', use: { ...devices['Desktop Chrome'], userAgent: `${devices['Desktop Chrome'].userAgent} (Checkly, https://www.checklyhq.com)` }, }, ], }) ``` Afterwards, you can reference this new project with the Checkly user-agent so that it's the one run in your checks: ```ts checkly.config.ts highlight={15} theme={null} import { defineConfig } from 'checkly' import { Frequency } from 'checkly/constructs' export default defineConfig({ projectName: 'WebApp E2E check suite', logicalId: 'webapp-e2e-check-suite', repoUrl: 'https://your.url.here', checks: { playwrightConfigPath: './playwright.config.ts', playwrightChecks: [ { name: 'My new check', logicalId: 'my-new-check', frequency: Frequency.EVERY_10M, pwProjects: ['chromium-checkly-agent'], // reference the new project locations: ['us-east-1', 'eu-central-1'], }, ], }, //test configuration }) ``` ## Filtering Google Analytics You might want to filter Checkly traffic in Google Analytics to prevent Checkly browser checks from skewing your results. If you are using Google Analytics 4 or newer, Checkly traffic is [already being excluded](https://support.google.com/analytics/answer/9888366?hl=en) as it is recognized as coming from a known bot-traffic source. ### Older Google Analytics versions If you want to filter Checkly traffic in older Google Analytics versions, here is one way to do it: 1. Add a UTM source tag to the URLs your requesting, i.e.: ```js theme={null} await page.goto('https://app.checklyhq.com/login?utm_source=monitoring') ``` 2. In Google Analytics, filter on campaign source. filter google analytics traffic on campaign source

filter google analytics traffic on campaign source

For detailed instructions, see [the Google Analytics docs on custom filters](https://support.google.com/analytics/answer/1033162#CustomFilters). Note that this will take some hours for this to take effect. ## CloudFlare Verified Bot Checkly is also registered with CloudFlare as a [verified bot](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/). If you are using CloudFlare's Web Application Firewall (WAF) to protect your site, you can also allow traffic from Checkly by adding a [custom WAF rule](https://developers.cloudflare.com/waf/custom-rules/) to enable traffic from verified bots. Specifically, the [Checkly verified bot](https://radar.cloudflare.com/bots/directory/checkly) is included in the [Monitoring & Analytics category](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/categories/). CloudFlare automatically syncs the latest Checkly IP addresses, so this approach can be easier to maintain than [IP range allowlisting](#ip-range-allowlisting). CloudFlare verified bot WAF rule

## Default Checkly user agent This is what Checkly sends as user agent: * API & Multistep checks: `Checkly/1.0 (https://www.checklyhq.com)`. * Browser checks: `Checkly, https://www.checklyhq.com`. Changing the user agent of a check will not change the browser or browser version, it will simply set a new string for user agent. # How we store data Source: https://www.checklyhq.com/docs/platform/data-storage Learn about how Checkly stores check run data Every check we run generates a certain amount of data. We call these the "results". Results are made up of various parts: * Metadata like timestamps, response times, fail/success flags, run location etc. * JSON or other response bodies, response headers, request timing data * Playwright Test traces and videos * Screenshot files * Run logs with debug or `console.log` statements. ## Data Retention Depending on your plan we keep the results data according to the following retention policy: | | Hobby | Starter | Team | Enterprise | | ----------------------------- | ------- | ------- | ------- | ---------- | | **Raw data retention** | 7 days | 7 days | 30 days | 180 days | | **Aggregated data retention** | 30 days | 30 days | 1 year | 25 months | Simply put, you can "scroll back" within the defined period for raw data and inspect all the details of each result. After that, we store aggregates of the relevant statistics like average, p95 and p99 response time and success ratio. # Dynamic Secret Scrubbing Source: https://www.checklyhq.com/docs/platform/dynamic-secret-scrubbing Scrub sensitive data from check results Dynamic secret scrubbing allows you to retrieve secrets at runtime from external vaults (like Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault) while automatically removing those sensitive values from logs, traces, screenshots with text, and report trees. This is particularly useful for organizations that prefer not to store secrets in Checkly and want to maintain their existing secret management infrastructure. Set environment variables with the `CHECKLY_SECRET_*` prefix and the runtime will detect and scrub those values from all check results. ## How it works Currently, dynamic secret scrubbing is only supported for Browser and Multi-Step Checks. Assign values to `process.env.CHECKLY_SECRET_*` in your check code: ```javascript theme={null} // Direct assignment process.env.CHECKLY_SECRET_API_KEY = 'your-secret-value' // From external sources process.env.CHECKLY_SECRET_PASSWORD = await getFromAzureKeyVault('db-password') process.env.CHECKLY_SECRET_TOKEN = await fetchFromVault('auth-token') ``` The runtime automatically detects these variables and scrubs their values from: * Check logs * Trace files * Screenshots containing text * Report trees ## Supported patterns ```javascript theme={null} // Bracket notation process.env['CHECKLY_SECRET_DATABASE_URL'] = connectionString // Direct assignment process.env.CHECKLY_SECRET_AUTH_TOKEN = token // Dynamic retrieval process.env.CHECKLY_SECRET_PAYMENT_KEY = await vault.get('payment-api-key') ``` ## Limitations * **Check types**: Only works in browser and multistep checks * **Value format**: Must be a string (empty strings, `null`, `undefined`, numbers, objects, and arrays are ignored) * **Size limit**: Values cannot exceed 128KB (\~128,000 characters) * **Pattern-based detection**: Scrubbing works by detecting code and output structures. Complex or unconventional Playwright usage may fall outside of this detection. Follow the documented patterns for reliable scrubbing. ```javascript theme={null} // This works ✅ const apiKey = process.env.CHECKLY_SECRET_API_KEY // This doesn't work ❌ const key = 'CHECKLY_SECRET_' + 'API_KEY' const apiKey = process.env[key] ``` ## Example usage ```javascript theme={null} import { test } from '@playwright/test' test('API call with scrubbed credentials', async ({ page }) => { // Set secrets at runtime process.env.CHECKLY_SECRET_API_TOKEN = await getTokenFromVault() process.env.CHECKLY_SECRET_USER_ID = await getCurrentUserId() // Use in your test - values will be scrubbed from results await page.request.post('/api/data', { headers: { 'Authorization': `Bearer ${process.env.CHECKLY_SECRET_API_TOKEN}`, 'X-User-ID': process.env.CHECKLY_SECRET_USER_ID } }) }) ``` ## Runtime compatibility This feature is available from runtime `2024.09` onwards. For private locations running older agent versions, contact support for access. # Groups Source: https://www.checklyhq.com/docs/platform/groups Organize your checks and apply shared configuration with Groups Groups help you organize your checks (e.g. by team or feature) and apply shared configuration such as API defaults, scheduling & location overrides, and other properties. Check group screenshot

Groups serve multiple purposes: they act as organizational folders, configuration templates, and execution units. Think of them as the filing system for your monitoring infrastructure that transforms scattered checks into a coherent system reflecting your application architecture and team responsibilities. You can populate a group by moving existing checks into it or by creating new checks directly within the group. By default, newly created check groups behave like folders, with no group-level configuration applied. To get started: ## Configuration Groups let you apply shared configuration to standardize how checks behave. Group-level configurations override check-level settings when applied. ### Check Membership **Moving a check into a group:** If the group has group-level configuration defined, adding a check may change how it runs. Settings like API defaults, locations & scheduling, or retries & alerting can override the check's configuration. **Removing check from group:** Any group-level configuration will no longer apply, and the check will use its own configuration going forward. To prevent issues (e.g. broken references to group variables), the check will be automatically deactivated after being added to or removed from a group. Make sure to review its settings before reactivating. Using a set of shared defaults for API checks helps you manage checks that go to the same basic endpoint, share headers and other settings. ## API request defaults You can set defaults for many aspects of an API request, see the separate paragraphs below. The following properties are still controlled at the check level: * HTTP method & body * response time limits #### Base URL Use the **base url** to define the shared protocol, domain and path for URLs in your group's API checks. This works as follows: 1. Set the base URL, i.e. `https://api.example.com/v1` 2. The base URL is now available in the `{{GROUP_BASE_URL}}` variable 3. In your API check, append path and query params, i.e. `{{GROUP_BASE_URL}}/customers?page=1` #### Headers & query parameters Any headers and query parameters defined in the API check defaults are injected into each API check. > Headers and query parameters at the check level override those at the group level with the same name. An example: 1. At the group level, you define the header `X-Custom:` with value `123` 2. At the API check level, you define the same header `X-Custom` just with the value `abc` 3. Checkly will call your API with the `X-Custom: abc` header #### Basic Auth Use the basic auth username and password to inject it into each API check in the group. #### Assertions Any [assertions](/docs/detect/assertions) are injected into each API check's assertion list. Use this to always assert common response codes or headers for your API checks. #### Setup & teardown scripts You can add [setup and teardown scripts](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown) at the group level as well as the individual group level. This is the execution flow: 1. Group level setup script 2. Check level setup script 3. **Execute your API check** 4. Check level teardown script 5. Group level teardown script A common scenario for having two levels of setup scripts is the following: 1. Group level setup: Fetch a token from a common authentication endpoint. 2. Check level: prep some specific test data for this individual check. ### Variables You can set both variables and secrets for a Group of checks. For browser and multistep checks, you can set variables at the check level. See [browser check variables and secrets](/docs/platform/variables) for more details. Secrets are fully supported starting with runtime version 2024.09 and later. For [Private Locations](/docs/platform/private-locations/overview), secrets are available in agent version `3.3.4` and later, and for the [CLI](/docs/cli/overview), in version `4.9.0` and later. #### Variable hierarchy As checks are scheduled, Checkly merges the check, group and global environment variables into one data set and exposes them to the runtime environment. During merging, any check variable with the same name as a global or group variable **overrides that variable.** Or, in other words: **check** variables trump **group** variables trump **global** variables. You can make use of this by providing a default value for a specific variable at the global or group level, but allow that variable to be overridden at the group level or check level. ### Scheduling & Locations * **Locations:** Override location settings for all checks in the group with selected public or private locations * **Scheduling strategy:** Override scheduling strategy (parallel, round-robin) for all checks in the group * **Frequency:** Set a default scheduling interval for new checks created in the group ### Retries & Alerting * **Retries:** Configure retry strategy for failed checks (fixed, single, linear, exponential) * **Alert settings:** Set up alert channels for the group - email, SMS, Slack, webhooks, etc. ### Testing & Runtimes * **Testing:** Run checks in this group as E2E tests locally or from CI/CD pipelines * **Runtimes:** Set the JavaScript runtime environment for browser checks and scripts if different from account default ## How Grouped Checks Execute Understanding how grouped checks run is important for sophisticated monitoring setups: 1. **Individual scheduling:** Checks are scheduled as individual checks, not "as a group" 2. **Group triggers:** Calling a "group run" using CLI or GitHub integration runs all checks in the group 3. **No group state:** The group itself does not have an explicit runtime state 4. **Individual results:** No results or metrics are collected at the group level 5. **Independent timing:** Checks in a group maintain their individual scheduling settings Groups are primarily configuration containers that provide shared settings and organizational structure. This approach ensures flexibility while enabling powerful configuration management at scale. ## Flexible Configuration Strategy Groups can serve different organizational needs: * **Strict templates:** Enforce consistency through shared configurations * **Loose containers:** Provide structure without imposing shared settings * **Team boundaries:** Enable clear ownership and delegation of monitoring responsibilities * **Scaling mechanism:** Manage complexity through logical decomposition as your infrastructure grows ## Mixing Checks and Monitors in a Group When you purchase a plan in Checkly, it includes access to specific monitoring features. As shown on our [pricing page](https://www.checklyhq.com/pricing), feature availability can differ between: * Uptime monitors (e.g. URL, TCP, DNS) * Synthetic checks (e.g. API, Multistep, Browser, Playwright Check Suites) This means monitors and checks on the same plan may not all have access to the same features. For example: A Browser check might support `parallel` scheduling, while a TCP monitor only allows for `round-robin` scheduling. Groups let you organize any type of checks/monitors together and define [shared settings](/docs/platform/groups#configuration). If those shared settings aren't supported by all monitors in the group, Checkly will throw a validation error asking you to either adjust the configuration or remove the incompatible monitor. If you're unsure how to proceed, don't hesitate to reach out to our [support team](mailto:support@checklyhq.com), we're happy to help. # IP Information & Versions Source: https://www.checklyhq.com/docs/platform/ip-information Understand Checkly's dual-stack IPv4 and IPv6 support, how address resolution is prioritized, and where to find the static IP ranges for allowlisting. ## IPv4 and IPv6 support Checkly supports both IPv4 and IPv6. We have a dual stack setup, meaning that we can reach both IPv4 and IPv6 addresses from our check runtime. Note that for most workloads (API checks, browser checks, etc.) the IPv6 stack is preferred over IPv4. This means that if your endpoint is reachable over IPv6, we will use that. If not, we will fall back to IPv4. You can actually influence this behaviour in a browser check by passing the `--host-resolver-rules` flag to a `chromium` instance. In the code example below, we force the browser to use IPv4, whereas the host `dual.tlund.se` is reachable over both IPv4 and IPv6. ```ts resolver.spec.ts theme={null} import { test, chromium } from '@playwright/test' test('Set host resolver to IPv4', async () => { const browser = await chromium.launch({ args: ['--host-resolver-rules=MAP dual.tlund.se 193.15.228.195'] }); const page = await browser.newPage(); await page.goto('http://dual.tlund.se/') }); ``` ## Static IP ranges All monitoring traffic from Checkly for all customers on all plans comes from a fixed set of source IPs. An up-to-date list can be requested unauthenticated through the API. See our [API spec](https://api.checklyhq.com/#/Static%20IPs) for more info. If you are looking to allowlist Checkly traffic, [see our docs on allowlisting](/docs/platform/allowlisting-traffic). # Manage secrets at scale Source: https://www.checklyhq.com/docs/platform/manage-secrets-at-scale Load secrets from an external secrets manager during deployment and scope them to the Checkly groups and checks that need them. Use an external secrets manager as the source of truth when you manage many credentials with Monitoring as Code. Load the values into the deployment environment, then deploy each secret at the narrowest Checkly scope that needs it. This prevents every check execution from receiving a large account-wide secret set while keeping secret values out of your repository and CI/CD configuration. * A Checkly Monitoring as Code project using a current version of the Checkly CLI. * A secrets manager that your CI/CD workload can authenticate to. * Permission to deploy the checks and groups that will use the secrets. * Runtime `2024.09` or later. Private Locations require agent `3.3.4` or later. ## Understand the names in these examples | Name | Meaning | | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `CHECKLY_API_KEY` | Required secret credential used by the Checkly CLI. Supply an API key for your Checkly account. | | `CHECKLY_ACCOUNT_ID` | Required identifier for the Checkly account the CLI deploys to. This value is not secret. | | `CHECKLY_SECRET_` | Deployment environment variable read by `secret('')`. You define `` for each secret in your project. | | `PAYMENTS_API_KEY`, `PAYMENTS_ADMIN_TOKEN`, `payments-api-key`, and `payments.example.com` | Fictional application values used only to demonstrate group and check scoping. Replace them with the credentials, names, and URLs used by the service you monitor. | ## How the deployment works ```mermaid theme={null} flowchart LR A["CI/CD job"] -->|"short-lived identity"| B["External secrets manager"] B -->|"required values"| A A -->|"CHECKLY_SECRET_*"| C["Checkly CLI deploy"] C -->|"secret: true"| D["Scoped Checkly secrets"] D --> E["Relevant group and check runs"] ``` The secret value exists in the deployment process long enough for the Checkly CLI to synthesize and deploy the construct. Checkly then encrypts it and stores it at the configured group or check scope. `secret()` is a deployment-time environment lookup, not a persistent reference to your external secrets manager. Rotating a value in the external manager does not update Checkly until you deploy again. ## Choose the narrowest scope Checkly [merges variables when a check runs](/docs/platform/variables#variable-hierarchy). Check-level values override group-level values, which override account-level values. | Scope | Use it for | | ------- | --------------------------------------------------------------------- | | Account | Values genuinely needed by almost every check in the account. | | Group | Credentials shared by one service, application, environment, or team. | | Check | Credentials used by one check only. | Each check or group can define up to 100 environment variables. If one deployment project spans many teams or systems, consider splitting it into independently deployed projects so each deployment identity can access only its own secrets. ## Read secrets during deployment The `secret()` helper reads `CHECKLY_SECRET_` from the environment and stops with an error when the value is missing. For example, `secret('PAYMENTS_API_KEY')` reads `CHECKLY_SECRET_PAYMENTS_API_KEY`. ```typescript secrets.ts theme={null} import { secret } from 'checkly/util' export const PAYMENTS_API_KEY = secret('PAYMENTS_API_KEY') export const PAYMENTS_ADMIN_TOKEN = secret('PAYMENTS_ADMIN_TOKEN') ``` Your external secrets manager can use different names. Map its identifiers to the `CHECKLY_SECRET_*` names expected by the project when you fetch the values in CI/CD. Every CLI command that loads these constructs needs the corresponding environment values, including `checkly test` and `checkly deploy`. Never provide production secrets to pull request workflows that run untrusted code. ## Deploy a group-level secret ```typescript __checks__/payments-group.check.ts theme={null} import { ApiCheck, CheckGroupV2, Frequency } from 'checkly/constructs' import { PAYMENTS_API_KEY } from '../secrets' const paymentsGroup = new CheckGroupV2('payments', { name: 'Payments', activated: true, muted: false, frequency: Frequency.EVERY_10M, locations: ['us-east-1'], environmentVariables: [ { key: 'PAYMENTS_API_KEY', value: PAYMENTS_API_KEY, secret: true, }, ], }) new ApiCheck('payments-health', { name: 'Payments health', group: paymentsGroup, request: { method: 'GET', url: 'https://payments.example.com/health', headers: [{ key: 'Authorization', value: 'Bearer {{PAYMENTS_API_KEY}}' }], }, }) ``` `{{PAYMENTS_API_KEY}}` is resolved by Checkly when the API check runs. It does not read a secret from your CI/CD provider or external secrets manager. ## Deploy a check-level secret Define a secret directly on the check when no other check needs it: ```typescript __checks__/payments-admin.check.ts theme={null} import { Frequency, ApiCheck } from 'checkly/constructs' import { PAYMENTS_ADMIN_TOKEN } from '../secrets' new ApiCheck('payments-admin-health', { name: 'Payments admin health', activated: true, muted: false, frequency: Frequency.EVERY_10M, locations: ['us-east-1'], environmentVariables: [ { key: 'PAYMENTS_ADMIN_TOKEN', value: PAYMENTS_ADMIN_TOKEN, secret: true, }, ], request: { method: 'GET', url: 'https://payments.example.com/admin/health', headers: [{ key: 'Authorization', value: 'Bearer {{PAYMENTS_ADMIN_TOKEN}}' }], }, }) ``` Inside Browser, Multi-Step, Playwright, and API setup or teardown scripts, read the deployed value with `process.env.PAYMENTS_ADMIN_TOKEN` instead. ## Treat the deployment as authoritative For checks and groups managed by a Checkly CLI project, the construct is the desired state: * Changing a declared secret manually in the Checkly UI is reverted by the next deployment. * Adding an undeclared group- or check-level secret in the UI removes it on the next deployment. * Removing a declared secret in the UI recreates it on the next deployment. Manage both the secret declaration and its scope in code. Manage the value in the external secrets manager. ## Secure the deployment job * Prefer workload identity federation or OIDC over long-lived cloud credentials. * Grant the deployment identity read access only to the secrets required by that project. * Fetch secrets only in trusted deployment workflows. Do not expose production secrets to workflows triggered from forks or untrusted pull requests. * Avoid printing command output that contains secret values. Mask dynamically retrieved values according to your CI/CD provider's guidance. * Expose secrets only to the step that runs `npx checkly deploy` where your workflow system allows it. * Split large projects along team or service ownership boundaries to reduce the number of secrets available to one deployment process. ## Deploy from GitHub Actions Use the guide for your secrets manager: * [AWS Secrets Manager](/docs/integrations/ci-cd/github/secrets/aws-secrets-manager) * [Google Secret Manager](/docs/integrations/ci-cd/github/secrets/google-secret-manager) * [Azure Key Vault](/docs/integrations/ci-cd/github/secrets/azure-key-vault) * [HashiCorp Vault](/docs/integrations/ci-cd/github/secrets/hashicorp-vault) For Browser and Multi-Step checks that must retrieve credentials while the check is running, see [Dynamic Secret Scrubbing](/docs/platform/dynamic-secret-scrubbing). Runtime retrieval is a separate pattern with different check-type support. # Managing False Positives Source: https://www.checklyhq.com/docs/platform/managing-false-positives Managing False Positives in Checkly Reliable monitoring depends on delivering accurate alerts, ensuring that you're notified—and if it comes to it, woken up in the middle of the night—only when there’s a real issue. However, false positives can happen for various reasons. In this guide, we focus on one of the most common causes: unpredictable network behavior. The public internet, with its millions of networks and nodes, can be unpredictable, making occasional hiccups unavoidable. A harmless glitch in the network connection between Checkly and your check target might cause your site to be flagged as slow or unreachable, even when it's running perfectly fine. Thankfully, many of these are just isolated incidents that resolve on their own, and there are ways to protect your monitoring setup from them. At Checkly, we aim to help you achieve the lowest false positive rate in the market, ensuring you are alerted only when it truly matters. Read on to learn more about common causes of performance variations and strategies to minimize the false positives they can cause. ## Common Causes of Performance Variations When setting up your monitoring strategy, it's important to consider a few common factors that can affect networking performance: * **Network Latency:** The time it takes for data to travel between points. This can vary due to distance, routing, or congestion. * **Packet Loss:** Sometimes data packets don’t reach their destination and need to be resent, leading to delays. * **Server Response Time:** Servers might occasionally take longer to respond, or load balancers and backend services could be at capacity. * **Infrastructure Issues:** Temporary issues like routing errors, server load spikes, or scheduled maintenance can also impact performance. * **Security-Related Blockages:** Firewalls, DDoS protection, or other security measures might prevent Checkly’s monitoring traffic from reaching your servers if flagged as suspicious. This can disrupt checks and result in false positives. While these factors are generally well-managed, they can't be completely avoided. However, you can take steps to make your monitoring setup more resilient. ## Best Practices to Minimize False Positives False positives caused by network variations can lead to unnecessary alerts, but you can keep them under control by following these best practices: ### Use Retries Retries are your first line of defense against transient issues. By setting up retries, your checks can automatically try again if the first attempt fails because of a temporary glitch. This reduces the impact of brief disruptions and helps prevent false positives. While you might want to adjust based on specific use cases, we suggest implementing retries for checks with a high escalation impact as a standard best practice. Learn more about implementing retries: [Retries at Checkly](https://www.checklyhq.com/docs/communicate/alerts/configuration#retries). ### Optimize Your Regional Monitoring Running checks from a specific region is important when you need to monitor your app’s performance in that area. To identify localized failures, implementing retries can help confirm whether the issue is temporary or persistent. However, if you want to understand whether an issue is more widespread, running checks from multiple regions can help you determine if it’s a broader problem with your app. Depending on your use case, we recommend running checks from at least two different regions to more accurately differentiate between a regional network issue and a genuine application problem. Learn more about setting up multi-location checks: [Global Location and Scheduling Strategies](/docs/concepts/locations). ### Fine-Tune Your Alerting Adjust your alert settings to avoid unnecessary notifications. For example, you can require failures in multiple regions or after several retries before firing off an alert. This helps you avoid alerts for short-lived, isolated failures that don’t need immediate attention. A good starting point is to set alerts to trigger only after a check has failed at least twice (or after two intervals) and has failed in more than one region. Learn more about fine-tuning your alerts: [Alert Settings at Checkly](https://www.checklyhq.com/docs/communicate/alerts/configuration). ### Allowlist Checkly Traffic In some cases, firewalls, load balancers, or security solutions might inadvertently block Checkly's monitoring traffic, which could interfere with running checks successfully. This can happen if your system detects Checkly’s requests as suspicious or treats them as unwanted traffic. To prevent this, we recommend reviewing your security settings to ensure that Checkly’s IP ranges are included in your allowlist. You can find Checkly’s current IP ranges here: [Allowlisting & filtering traffic](/docs/platform/allowlisting-traffic#ip-range-allowlisting). ## If Issues Persist Use [Checkly's Network Diagnostic Tools](/docs/platform/network-diagnostic-tools) to inspect failed checks for issues like DNS errors, dropped packets, or failed handshakes. If that doesn't reveal the root cause, don't hesitate to reach out to [support@checklyhq.com](mailto:support@checklyhq.com), and our team will help investigate further. # Network Diagnostic Tools Source: https://www.checklyhq.com/docs/platform/network-diagnostic-tools Network Diagnostic Tools in Checkly When troubleshooting network issues like dropped packets or broken TCP handshakes you often require low-level visibility into what's happening on the network. That's where network diagnostic tools come in. They help you understand how data moves through a network and where things might be breaking down. Checkly provides: * **TCP dumps:** capture network traffic to inspect packet-level behavior * **Traceroute report:** trace the path your request takes across the internet * **ICMP monitors:** continuously monitor host reachability, latency, and packet loss Whether you're debugging a slow response or tracing a connection failure, these tools can help you get to the root of the problem. ## TCP dumps TCP dumps give you a detailed look at what's happening at the packet level when a request fails. They capture the packets exchanged between Checkly and your target, letting you analyze things like: * Did the TCP handshake complete? * Was the TLS handshake successful? * Were packets dropped, delayed, or malformed? This level of visibility is especially helpful for diagnosing flaky networking behavior. ### How to access TCP dumps are currently only available for API checks. For checks that fail due to network errors (e.g. DNS resolution issues, TCP handshake failures, connection timeouts), you'll find a *Download PCAP* button in the **Network Diagnostics** section of the check results page. This gives you a `.pcap` file with the raw packet data from that request. You can open this file using tools like Wireshark to inspect each packet in detail. ## Traceroute report Traceroute shows the path your request takes across the network and how each hop behaves. A report typically includes: * The sequence of network hops between Checkly and your target * Packet loss percentage per hop * Latency statistics (last, average, best, worst, standard deviation) This makes it especially useful for diagnosing: * Where along the route packets are being dropped * Which hops introduce unusual latency or instability ### How to access Traceroute is currently only available for API checks. On checks that fail due to network errors (e.g. DNS resolution issues, TCP handshake failures, connection timeouts), a traceroute report is displayed in the **Network Diagnostics** section of the check results page. The table shows each hop, its IP or hostname, along with metrics such as packet loss, latency, and jitter. ## ICMP monitors [ICMP monitors](/docs/detect/uptime-monitoring/icmp-monitors/overview) let you proactively detect network issues by continuously pinging a host. They measure round-trip latency and packet loss over time, making them useful for: * Verifying reachability of hosts that don't expose HTTP endpoints * Detecting network-level outages before they affect application-layer checks * Tracking latency and packet loss trends across regions Learn more about setting up ICMP monitors in the [ICMP monitor overview](/docs/detect/uptime-monitoring/icmp-monitors/overview) and [configuration](/docs/detect/uptime-monitoring/icmp-monitors/configuration) docs. # Agent Configuration Source: https://www.checklyhq.com/docs/platform/private-locations/agent-configuration Configure the Checkly Agent for a private location with environment variables for API keys, proxies, job concurrency, log level, and dependency caching. The Checkly Agent is a container that you need to deploy to run a Private Location in Checkly. The agent needs to be deployed on your infrastructure and executes checks on behalf of the Checkly application. For installation details, [check the getting started guide](/docs/platform/private-locations/overview). The Checkly Agent has several environment variables that can be configured: | Variable | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `API_KEY` | API key for the Private Location in which the agent will serve. The API key is shown in the web app once, when initially creating a Private Location. | | `HTTPS_PROXY` | HTTPS proxy configuration for the outbound connection to the Checkly API, used for agent management and monitoring. `https://user:password@127.0.0.1:8080` | | `HTTP_PROXY` | HTTP proxy configuration for the outbound connection to the Checkly API, used for agent management and monitoring. Used if the proxy server does not accept HTTPS connections. `http://user:password@127.0.0.1:8080` | | `JOB_CONCURRENCY` | (Default: 1, max: 10) Number of concurrent checks that are run by the agent. | | `LOG_LEVEL` | (Default: `INFO`) Set the log level of the agent. Can be one of `DEBUG`, `LOG`, `INFO`, `WARN` or `ERROR`. | | `USE_OS_DNS_RESOLVER` | When set to true, TCP monitors will resolve DNS using `getaddrinfo` C function, instead of using the network. This enables easier DNS resolution for internal services e.g. services running in the same Kubernetes cluster. | | `DEPENDENCY_CACHE` | (Default: `OFF`) Set to `CHECKLY_S3` to enable dependency caching for [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview). Caches installed dependencies between runs to speed up check execution. Learn more about [dependency caching](/docs/detect/synthetic-monitoring/playwright-checks/custom-dependencies#dependency-caching). | | `AGENT_HEALTH_PORT` | (Default: `8081`) Port for the HTTP [health probe endpoints](#health-probe-endpoints). | | `EXTERNAL_OBSERVABILITY` | Enabled by default. Set to `off` to disable [external observability](#external-observability) reporting of agent runtime errors to Checkly. | For example, you can add these variables to the standard docker run command like this: ```bash Terminal theme={null} docker run -e API_KEY="pl_...." -d checkly/agent:latest ``` Here are some more usage examples: 1. Set the `JOB_CONCURRENCY` to the appropriate value (1-10) based on your scaling calculations: ```bash Terminal theme={null} -e JOB_CONCURRENCY=10 ``` 2. Configure an HTTPS or HTTP proxy if one is required for your environment: ```bash Terminal theme={null} -e HTTPS_PROXY="https://user:password@127.0.0.1:8080" ``` Once the agent container is running, you can see it in a running state using the appropriate command from your container engine (typically `docker ps`). ```bash Terminal theme={null} CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 72ec5591f6b2 checkly/agent:latest "node index.js" 5 hours ago Up 5 hours lucid_shockley ``` 3. You can also check the logs of the new container to ensure it's up and running (typically `docker logs `). ```bash Terminal theme={null} [checkly-agent] Starting Consumer c7495186-6f1e-4526-b173-14ee9ad21775 [checkly-agent] No jobs. Waiting.. [checkly-agent] No jobs. Waiting.. [checkly-agent] No jobs. Waiting.. [checkly-agent] No jobs. Waiting.. [checkly-agent] No jobs. Waiting.. ``` ## Local Docker workflow You can easily install the Checkly Agent on your Macbook, Windows or Linux machine for local debugging using [Docker Desktop](https://docs.docker.com/desktop/). The installation procedure is the same as described above. To resolve locally running webapps or APIs, e.g. some project running on `http://localhost:3000` you need to use the [internal Docker host](https://docs.docker.com/desktop/networking/) `http://host.docker.internal:3000` to bridge to your `localhost` address. ## Updating the agent container New versions of the Checkly Agent are released regularly to [Docker Hub](https://hub.docker.com/r/checkly/agent/tags). To have access to the latest improvements, you can update the Checkly Agent containers. View our [changelog](/docs/platform/private-locations/change-log) to learn more about each version. ### Manual updates To manually upgrade your Checkly Agent containers, first pull the image: ```bash Terminal theme={null} docker pull checkly/agent:latest ``` You can then start a new Checkly Agent using the updated image. If you no longer have the API key, you can always create a new one in the UI by [rotating API keys](#rotating-api-keys). ```bash Terminal theme={null} docker run -e API_KEY="pl_...." -d checkly/agent:latest ``` Finally, you can clean up old Checkly Agent containers. First, run `docker ps` to find the names of the previous Agent containers. ```bash Terminal theme={null} CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 72ec5591f6b2 checkly/agent:latest "node index.js" 5 hours ago Up 5 hours lucid_shockley db0aa54baf78 checkly/agent:latest "node index.js" 6 days ago Up 5 hours quirky_lumiere ``` You can distinguish the old containers from the new ones by looking at the `CREATED` field. Then run `docker stop ` and `docker rm ` to remove the old Agent containers. ### Automatic updates Use caution when enabling automatic updates in production. This could automatically update to a version with breaking changes, and upgrade to an agent version supporting a later runtime. Since the agent is stateless, it can also be updated by replacing or updating the image in place. If you don't have an existing process for upgrading containers, an in-place upgrade is easiest as it keeps the previously defined environment variables. Breaking changes are communicated by increasing the major version of the image (e.g. from v1.3.8 to v2.0.0). Each version of the agent only supports specific runtimes. You can find a list of agent versions and corresponding runtimes [below](/docs/platform/private-locations/agent-configuration/#agent-version-and-runtimes). You can use the [Watchtower tool](https://containrrr.dev/watchtower/) to do an in-place upgrade of an agent container. Ensure you have sufficient agent capacity as the agent container will have a short outage as it is upgraded. As agent shutdowns are graceful, no running checks will be lost: ```bash Terminal theme={null} docker run --rm -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once ``` ## Rotating API keys API keys can be rotated as necessary in order to maintain good security practices or in case a key is lost or compromised. A Private Location can have two active API keys. To rotate the API keys: 1. Navigate to Private Locations in your Account Settings. 2. Click the edit icon on the Private Location you want to update. 3. There will only be a single API key in place on a new Private Location. Click Add new API key to add the second key. 4. The new API key will appear. Click the copy icon to copy it to your clipboard and save it somewhere secure. new key

5. Click Save to close the dialog. You can now see the trailing characters of the two defined API keys in the list of Private Locations. two keys

6. You now need to replace your Checkly agents. You can do this one-by-one or as a group, just make sure you always have enough agent containers up and running based on your workload. Using your container management tool, start new agents with the new API key: ```bash Terminal theme={null} docker run -e API_KEY="pl_...." -d checkly/agent:latest ``` 7. Make sure the new agents are running properly. You can check the agent count on the Private Locations page, or check your container logs. Remove the old agent containers: ```bash Terminal theme={null} docker stop ``` If you lose track of which agent containers are using the old API key, you can use the `docker inspect ` command and look for the `API_KEY` variable in the output. 8. Now that your new agents are in place, remove the old API key. Click the edit icon in the Private Location again. 9. Click the delete icon next to the old API key (verified by the shown trailing characters) then click the confirmation. 10. Click Save to close the dialog. You will now only see the new API key listed for the Private Location. ## Trusting Enterprise TLS Certificates Some enterprise environments may use internal certificates for TLS connections. These certificates need to be trusted by the agent; otherwise, the agent's TLS connection to the Checkly platform will fail. The agent container runs two processes that each make outbound TLS connections to Checkly: the main agent process and a Node.js worker. Each uses its own trust store, so you typically need to configure both. First, add your Certificate Authority's (CA) root certificate to the system's CA store. On Debian, and related Linux distributions, you can do this by copying your `*.crt` certificate to `/usr/local/share/ca-certificates/` and running `sudo update-ca-certificates`. This is required for the main agent process. Then, configure the Node.js worker to trust the certificate. Copy the certificate as a PEM file onto the host, then update the Docker run command to mount the certificate as a volume and pass the path to the certificate in the `NODE_EXTRA_CA_CERTS` environment variable. For a certificate stored at the path `~/certificate.pem`, the Docker run command will be: ```bash Terminal theme={null} docker run -v ~/certificate.pem:/checkly/certificate.pem -e NODE_EXTRA_CA_CERTS=/checkly/certificate.pem -e API_KEY="pl_...." -d checkly/agent:latest ``` Alternatively, you can tell the Node.js worker to accept unauthorized certificates with the environment variable `NODE_TLS_REJECT_UNAUTHORIZED=0`. This can be appended to your `docker run ..` command with the `-e` flag. ## Health Probe Endpoints The Checkly Agent exposes HTTP health probe endpoints on port `8081` by default. You can change this port using the `AGENT_HEALTH_PORT` environment variable. | Endpoint | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------- | | `GET /-/readiness` | Returns `200 OK` when the agent is ready to accept work. Returns `503 Service Unavailable` during shutdown. | | `GET /-/liveness` | Returns `200 OK` to indicate the agent process is running. | | `GET /health` | Returns `200 OK` as a general health check. | These endpoints are useful for configuring [Kubernetes readiness and liveness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) or any other container orchestration health checks. ## External Observability The Checkly Agent reports error-level events to Checkly to help detect crashes, bugs, and environment-related issues. This is enabled by default starting with agent v7.0.0. Only limited technical metadata is collected (e.g. error messages, stack traces, agent version, platform, go runtime information). Sensitive data such as API keys, environment variables, check configuration, URLs, headers, and response bodies is **never** sent. To disable external observability, set the `EXTERNAL_OBSERVABILITY` environment variable to `off`: ```bash Terminal theme={null} -e EXTERNAL_OBSERVABILITY=off ``` ## Agent Version and Runtimes Each Checkly Agent only supports specific [runtimes](/docs/platform/runtimes/overview/). This is to keep the container image at an acceptable size. When you change the runtime a check uses, you may also need to change to the corresponding agent version. Similarly, if you update the agent version to one using a different runtime you also need to update your checks to use the same runtime. | Agent version | Runtimes | | ------------- | ---------------- | | 8.0.0 | 2026.04 | | 7.0.0\* | 2026.04, 2025.04 | | 6.0.0 | 2025.04, 2024.09 | | 5.0.0 | 2025.04 | | 4.0.0 | 2024.09 | | 3.5.0 | 2024.09 | | 3.4.0 | 2024.02 | | 3.2.0 | 2023.09 | | 2.0.0 | 2023.02 | | 1.3.9 | 2022.10 | \*If you're upgrading from runtime `2024.09` to `2026.04`, use the `7.0.2-migrate-202409-202604` migration image. It ships all three intermediate runtimes (`2024.09`, `2025.04`, `2026.04`) so you can migrate your checks incrementally. As each agent version only supports specific runtimes, we recommend pinning agent versions used in production. ## Troubleshooting Container runtimes drop `NET_RAW` by default. Traceroute and Packet Capture need raw sockets, so the Checkly Agent needs this capability explicitly granted. **How to fix**: In **Kubernetes**, update your pod spec or Helm values: * `allowPrivilegeEscalation=true` — prevents Kubernetes from setting the `no_new_privs` bit on the container process, which would otherwise block file capabilities (like `cap_net_raw+ep`) from taking effect. * `capabilities.add=["NET_RAW"]` — adds `CAP_NET_RAW` to the container's bounding and permitted capability sets, allowing the kernel to open the raw sockets (`SOCK_RAW`). ```yaml theme={null} securityContext: allowPrivilegeEscalation: true capabilities: add: ["NET_RAW"] ``` In **Docker**: ```bash theme={null} docker run --cap-add=NET_RAW ghcr.io/checkly/agent:latest ``` Docker does not set `no_new_privs` by default, so `--cap-add` is sufficient. If you run with `--security-opt no-new-privileges`, file capabilities on binaries will be blocked — remove that flag to allow `NET_RAW`. `NET_RAW` only grants permission to create raw sockets. It does not escalate broader container privileges. # Agent Images Source: https://www.checklyhq.com/docs/platform/private-locations/agent-images Compare the Checkly Agent images, Standard, Dev, and Uptime, and learn the image tag format for pulling the variant and version you need. Checkly provides different agent images: * [Standard](#standard-image): The full agent image, supporting all check types * [Dev](#dev-image): The standard image with build tools for compiling npm packages with native dependencies * [Uptime](#uptime-image): A lightweight image for running uptime monitors only ## Image Tag Format Image tags define which agent variant and version you run. ```bash theme={null} # Standard docker pull checkly/agent:X.Y.Z # Dev (with build tools) docker pull checkly/agent-dev:X.Y.Z # Uptime docker pull checkly/agent-uptime:X.Y.Z ``` Use versioned tags for reproducible deployments. Floating tags (like `:latest`) may change between pulls. Browse available versions on [Docker Hub](https://hub.docker.com/r/checkly/agent/tags). ## Standard Image The standard image (`checkly/agent:X.Y.Z`) is the default and recommended option for most setups. It supports the full range of Checkly monitors: * [Synthetic checks](/docs/detect/synthetic-monitoring/overview): API, Multistep, Browser and Playwright Check Suites * [Uptime monitors](/docs/detect/uptime-monitoring/overview): URL, DNS, TCP, ICMP Use the standard image unless you have a specific need for the dev or uptime variants. ## Uptime Image The uptime image (`checkly/agent-uptime:X.Y.Z`) is a lightweight version of the agent designed for [uptime monitoring](/docs/detect/uptime-monitoring/overview) only. It supports URL, DNS, TCP and ICMP monitors. ## Dev Image If any of the checks you run on Private Locations rely on npm packages with native code, those packages need to be compiled during installation. The standard agent is optimized for size and doesn’t ship with build tools. In those cases, you’ll need to use the dev image instead. #### When to Use the Dev Image Use the dev image (`checkly/agent-dev:X.Y.Z`) when your checks require npm packages with native dependencies that need compilation. Common examples include: * `sqlite3` - SQLite database * `zookeeper` - Apache ZooKeeper client If your checks only use pure JavaScript packages, the standard runtime image is recommended for its smaller size and faster startup. #### What's Included in the Dev Image The dev agent works exactly like the standard agent, but includes the build tools required to compile native modules. It adds the following to the standard image: | Tool | Purpose | | -------------------- | ------------------------- | | `gcc`, `g++`, `make` | Compile native extensions | | `python3` | Required by node-gyp | These tools enable compilation of native Node.js modules during `npm install`. #### FAQ Yes, the dev image is production-ready. It contains the same runtime as the standard image, plus build tools. The only tradeoff is a larger image size. No. Playwright and its dependencies are pre-installed in both variants. You only need the dev image if your check code imports npm packages with native dependencies. If `npm install` fails with errors about `node-gyp`, `python`, `gcc`, or "compilation failed", you likely need the dev image. # Autoscaling Source: https://www.checklyhq.com/docs/platform/private-locations/autoscaling Autoscale Checkly Agents in a Private Location with KEDA based on queued and in-flight check runs. Scale Checkly Agent pods automatically in relation to live load. This page covers the KEDA-based recipe; for static capacity planning, see [Scaling and Redundancy](/docs/platform/private-locations/scaling-redundancy). * Prometheus V2 metrics are being ingested for your account — the only source for this gauge. See [Exporting Metrics & Data via Prometheus V2](/docs/integrations/observability/prometheus-v2/). * Checkly Agents are deployed via the [Checkly agent Helm chart](https://github.com/checkly/helm-charts/tree/main/charts/agent) (or an equivalent `Deployment`). See [Kubernetes Deployment](/docs/platform/private-locations/kubernetes-deployment). * [KEDA](https://keda.sh) is installed in the cluster. ## The signal Checkly exposes the `checkly_private_location_check_runs` gauge through the Prometheus V2 exporter. Filtered by `state` and a `private_location_slug_name`, it provides the count of pending and currently-executing check runs in a single Private Location — the signal you drive replica count from. The relevant `state` values are: * `queued` — the check run has been scheduled but not yet picked up by an agent. * `inflight` — the check run is currently being executed by an agent. The gauge is aggregated on a \~1 minute interval, so checks that start and finish within that window may be excluded — their impact on Private Location capacity is negligible. ## KEDA `ScaledObject` The `ScaledObject` below provides sensible defaults — adjust the bounds and scaling behavior to match your check workload. ```yaml theme={null} apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: checkly-agent-autoscaler spec: scaleTargetRef: namespace: name: minReplicaCount: 2 maxReplicaCount: 10 advanced: horizontalPodAutoscalerConfig: behavior: scaleUp: policies: - type: Pods value: 1 periodSeconds: 60 scaleDown: selectPolicy: Min policies: - type: Pods value: 1 periodSeconds: 60 triggers: - type: prometheus metadata: serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090 metricName: checkly_private_location_check_runs threshold: "1" # Match the agent's JOB_CONCURRENCY. query: sum(checkly_private_location_check_runs{state=~"queued|inflight", private_location_slug_name=""}) ``` The query is scoped to a single Private Location by `private_location_slug_name`, so create one `ScaledObject` per Private Location. If you deploy agents with the [Checkly agent Helm chart](https://github.com/checkly/helm-charts/tree/main/charts/agent), template the `ScaledObject` alongside your chart values so the autoscaler ships with the deployment. For a Prometheus instance outside the cluster, add an [`authenticationRef`](https://keda.sh/docs/latest/scalers/prometheus/) pointing at a `TriggerAuthentication` resource with the appropriate credentials. ## How many pods you'll get KEDA queries Prometheus on its polling interval and turns the result into a target pod count. With `threshold: "1"`, that target is roughly the number of queued plus in-flight check runs — one pod per check. The pod count is then kept within `minReplicaCount` and `maxReplicaCount`. For example, with `threshold: "1"`, `minReplicaCount: 2`, `maxReplicaCount: 10`: | Queued + in-flight check runs | Resulting pods | | ----------------------------- | -------------- | | 0 | 2 (idle floor) | | 1 | 2 | | 3 | 3 | | 7 | 7 | | 20 | 10 (capped) | ## Tuning the bounds * **`threshold`** — set it to match the agent's `JOB_CONCURRENCY`. The default `JOB_CONCURRENCY` is `1`, so leave `threshold: "1"`. A higher value packs more checks per pod and can cause scheduling delays for long-running checks. * **`minReplicaCount`** — keep at `2` or higher so a single agent failure doesn't take the Private Location offline. See [Scaling and Redundancy](/docs/platform/private-locations/scaling-redundancy). * **`maxReplicaCount`** — must exceed your expected peak queued + in-flight check runs. If the cap is too low, queued check runs accumulate above it and are dropped after the 6-minute queue TTL. If you set `minReplicaCount: 0` to scale to zero when idle, [`cooldownPeriod`](https://keda.sh/docs/latest/reference/scaledobject-spec/#cooldownperiod) becomes important — it controls how long KEDA waits after the trigger goes inactive before scaling the deployment down to zero. ## Graceful termination In-flight checks on a terminating pod are rerun on another agent after a 300-second timeout. Set `terminationGracePeriodSeconds` above this on the agent pod spec so an evicted pod has room to drain before `SIGKILL`: ```yaml theme={null} spec: template: spec: terminationGracePeriodSeconds: 330 # Set to your longest-running check type; up to 1800 for Playwright Check Suites. ``` Maximum runtime by check type: | Check type | Maximum runtime | | ---------------------- | --------------- | | API, TCP, DNS, ICMP | 30 seconds | | Browser | 4 minutes | | Multistep | 4 minutes | | Playwright Check Suite | 60 minutes | ## Verify 1. Confirm KEDA created the HPA and is reading the metric: ```bash theme={null} kubectl get scaledobject,hpa -n ``` 2. Probe the signal directly: ``` sum(checkly_private_location_check_runs{state=~"queued|inflight", private_location_slug_name=""}) ``` 3. Schedule a burst of checks against the Private Location and watch the replica count climb toward `maxReplicaCount`, then settle back to `minReplicaCount` once the burst clears. ## See also * [Scaling and Redundancy](/docs/platform/private-locations/scaling-redundancy) * [Exporting Metrics & Data via Prometheus V2](/docs/integrations/observability/prometheus-v2/) * [Kubernetes Deployment](/docs/platform/private-locations/kubernetes-deployment) # Checkly Agent Changelog Source: https://www.checklyhq.com/docs/platform/private-locations/change-log Release history for the Checkly Agent For major release announcements and summaries, see Checkly's [product changelog](https://feedback.checklyhq.com/changelog?c=Checkly+Agent), filtered by `Checkly Agent`. Learn more about: * updating to the latest version in [updating the agent container](/docs/platform/private-locations/agent-configuration#updating-the-agent-container) * supported runtimes for each agent version in [agent version and runtimes](/docs/platform/private-locations/agent-configuration#agent-version-and-runtimes) ## Full release history * Internal errors of the agent's Node.js component (which runs browser, multistep and Playwright checks) are now reported to the private location's Errors tab, like the Go component's already were * Fixed a bug that prevented pcap files from being cleaned up * Expose `CHECKLY_TRACE_ID` to API check setup & teardown scripts * Playwright Check Suites: add support for Checkly Traces * Update various dependencies to fix vulnerabilities * Uptime checks can run with response time limits up to 45s (TCP, API and URL), enabled per account * gRPC checks support server-streaming methods, plus improved encoding for advanced schema types * Browser checks can simulate constrained bandwidth * API check responses render image previews * Playwright Check Suites: network and console entries are extracted from traces, and trace retention now follows your account setting * Fixes: pnpm workspace caching on repeated runs, empty Playwright video assets, artifact upload reliability, multi-step method timeouts, and SSL hostname handling * Update various dependencies to fix vulnerabilities * Support for running gRPC, traceroute, and SSL checks * Always scrub potentially sensitive headers (Authorization, x-api-key, cookie, etc) even if they don’t contain secrets * Downgrade one of the dependencies that prevented the agent from running on older CPU hardware * Update various dependencies to fix vulnerabilities * Add `--refresh-cache` functionality * Fixed a bug that caused agent health and readiness probes to fail during graceful shutdown. * Update vm2 dependency: 3.11.3 → 3.11.5. * Fixed a bug that prevented users of pnpm workspaces to run their PWCS on private locations in certain scenarios. * Test failures and setup errors are now surfaced in the user-visible log alongside the existing report. * Added support for cancelling running Playwright check suites * Playwright check suites can run in different NodeJs or Bun versions (22, 24, 26 or Bun 1.3) * Improved Playwright browsers installation * Playwright Check Suites now cache Playwright browsers across runs in the same Private Location Agent * Upgraded dependencies such as `vm2` 3.10.5 → 3.11.3 and `axios` 1.15.0 → 1.16.1 * Patched Checkly's `markCheckAsDegraded` helper when used with soft assertions * [Global timeouts](https://playwright.dev/docs/test-timeouts#global-timeout) now apply and surface correctly in Playwright Check Suites * Improved scheduling for long-running Playwright Check Suites * Resolved several dependency vulnerabilities in the private location image Latest agent image with [runtime 2026.04](/docs/platform/runtimes/runtime-specification/#2026-04) only. * Previous versions bundled two runtimes to support gradual upgrades * This could surface vulnerabilities from the older runtime in security scans * From v8 onward, only the latest runtime is included **Upgrade path** * Already on `2026-04`: upgrade directly * On older runtimes: use the [migration image](#v7-0-2) (`7.0.2-migrate-202409-202604`) to move to `2026-04`, then upgrade to v8 Patches vulnerabilities in dependencies bundled with the [2026.04 runtime](/docs/platform/runtimes/runtime-specification/#2026-04). Fix-forward updates only. Updated packages: * `@azure/identity` 4.13.0 → 4.13.1 * `@clerk/testing` 1.14.2 → 1.14.3 * `@opentelemetry/exporter-metrics-otlp-grpc` 0.212.0 → 0.215.0 * `@xmldom/xmldom` 0.9.8 → 0.9.10 * `axios` 1.13.5 → 1.15.0 * `imapflow` 1.2.10 → 1.3.1 * `lodash` 4.17.23 → 4.18.1 * `prisma` 7.4.1 → 7.8.0 * `protobufjs` 8.0.0 → 8.0.1 * Includes security patches for runtime `2026-04` * Migration image (`7.0.2-migrate-202409-202604`) available: Bundles runtimes `2024-09`, `2025-04`, and `2026-04`, allowing you to upgrade from older runtimes to `2026-04` * [ICMP monitors](/docs/detect/uptime-monitoring/icmp-monitors/overview) no longer require full root privileges to run * Fixed broken runtime dependencies in agent containers. Incorrect symlinks for the `2025-04` and `2026-04` runtimes caused dependency resolution failures (affecting agent v6.3.2 and v7.0.0) * Fixed connectivity issues when IPv6 is disabled. The agent now falls back to IPv4 automatically * API checks and URL monitors now capture [packets and traceroutes](/docs/platform/network-diagnostic-tools/) on network failures * Bun is now supported as a package manager in [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview/) * Degraded status now correctly appears in the [test reporter](/docs/detect/testing/playwright-reporter/) header * Added support for running [ICMP](/docs/detect/uptime-monitoring/icmp-monitors/overview) and [DNS monitors](/docs/detect/uptime-monitoring/dns-monitors/overview) on Private Locations * Introduced the [uptime agent image](/docs/platform/private-locations/agent-images#uptime-image), a lightweight version of the agent for running uptime monitors (URL, DNS, TCP, ICMP) only * Added [external observability](/docs/platform/private-locations/agent-configuration#external-observability) for Private Location agents to capture runtime errors * Error-level events can be reported to Checkly to help detect crashes, bugs, and environment-related issues * Only limited technical metadata is collected (error message, stack trace, agent version, platform, Go runtime) * Sensitive data such as API keys, environment variables, check configuration, URLs, headers, and response bodies is never sent * Enabled by default, can be disabled with `EXTERNAL_OBSERVABILITY=off` * Support for runtimes `2026.04` and `2025.04` * Improved error logging for failed Playwright test report uploads, including the report file size * Increased the maximum artifact upload size from 75 MB to 150 MB The agent now exposes HTTP health probe endpoints: * **GET /-/readiness**: Returns `200 OK` when the agent is ready, `503 Service Unavailable` during shutdown * **GET /-/liveness**: Returns `200 OK` * **GET /health**: Returns `200 OK` Health endpoints run on port 8081 by default, configurable via the `AGENT_HEALTH_PORT` environment variable * Added [agent-dev image](https://www.checklyhq.com/docs/platform/private-locations/agent-images/) including build tools to support building native npm packages. Going forward, the dev variant is published as a separate image rather than a tag on the main agent image * Improved secret scrubbing speed * Improved error logging for hanging or long running Playwright Check Suite jobs Ensured `devDependencies` are not installed in the agent image * Added support for a [dev image](https://www.checklyhq.com/docs/platform/private-locations/agent-images/) including build tools to support building native npm packages * Improved internal logging and metrics for [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) * Browser and Multistep checks using runtime 2025.04 now automatically scrub the `Authorization` header from requests Added Checkly environment variables and configuration to [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview). See the docs on [Playwright Check Suites environment variables](/docs/detect/synthetic-monitoring/playwright-checks/environment-variables): * Adds support for `CHECKLY` to all checkly triggered runs: scheduled checks and test sessions triggered through either `npx checkly test`, `npx checkly trigger` or `npx checkly pw-test` * Includes the environment variable `CI` as true only for test session triggered runs Added Playwright test metrics to the check overview page, including **Test performance** and **Test statuses** graphs Minor patches on reporting Improved and centralized secret scrubbing for more reliable removal of sensitive data Added support for `CHECKLY_SECRET_*` environment variables. Secrets containing special characters (e.g. `@`, `%`, `!`, `+`) are now reliably scrubbed from traces, including when they appear in encoded form in URLs or request payloads. Learn more in our documentation on [Dynamic Secret Scrubbing](/docs/platform/dynamic-secret-scrubbing/) Fixed an issue where checks could fail behind proxies during dependency installation because proxy-related environment variables were not consistently available * Added runtime secret detection for Browser and Multistep checks (enabled for selected accounts) * See the [product changelog](https://feedback.checklyhq.com/changelog/checkly-agent-610-playwright-check-suites-and-more) for details on changes from version 6.0.1 to 6.1.0 Internal improvements and fixes related to networking and request handling # Kubernetes Deployment Source: https://www.checklyhq.com/docs/platform/private-locations/kubernetes-deployment Deploy the Checkly Agent to a private location on Kubernetes using the official Helm chart for production-grade, orchestrated agent deployments. We advise running any production-grade Checkly Agent deployments on a container orchestrator like Kubernetes. To help you get started, we provide a [public Helm chart](https://github.com/checkly/helm-charts) for deploying the agent. ## Prerequisites * You've [created a Private Location](/docs/platform/private-locations/overview) and have an API key * You have a running Kubernetes cluster * [Helm](https://helm.sh/) is installed ## Deploying the Checkly Agent with Helm See the [Checkly Helm Charts README](https://github.com/checkly/helm-charts/tree/main/charts/agent#readme) for installation instructions and configuration options. If you run into any issues, feel free to open an issue on the repository or reach out via the [Community Slack](https://join.slack.com/t/checklycommunity/shared_invite/zt-2qc51mpyr-5idwVD4R4izkf5FC4CFk1A). # Getting started with Private Locations Source: https://www.checklyhq.com/docs/platform/private-locations/overview Monitor internal systems and test from custom locations using Checkly Private Locations. Run checks from within your own infrastructure with the lightweight Checkly Agent. **Monitoring as Code**: Learn more about the [Private Location Construct](/docs/constructs/private-location). **Owner** or **Admin** permissions are required to create, edit, and delete Private Locations. A Private Location is a monitoring location that you manage by simply deploying a lightweight Checkly Agent. Running a check from a Private Location allows you to: * **Monitor internal systems**: Test the performance and reliability of applications and APIs that are only accessible from within your network (e.g., development environments, intranet tools) * **Test from anywhere**: Test the performance and reliability of applications and APIs from different parts of the world. Install your Checkly Agent(s) for location-based performance insights, that we don't cover with our [Global Locations](/docs/concepts/locations) Configure your checks to use your Private Location and that's it. As long as you have at least one operational Checkly Agent, checks will run in your Private Location. [Adding more agents](/docs/platform/private-locations/scaling-redundancy) will distribute the load and improve resilience automatically. ## Requirements Here are the requirements before you get started: * A container runtime (we test using Docker, but other runtimes should work) * Outbound internet access for the Agent to [https://agent.checklyhq.com](https://agent.checklyhq.com) (proxy configuration is supported) * Access to your API or browser-based applications and services from the Private Location network * Owner or Admin permissions (required to create, edit, and delete Private Locations) ## Configuring a Private Location 1. Navigate to the [Private Locations page](https://app.checklyhq.com/private-locations) and click "New private location" private locations screen

2. Provide a name and icon for the new private location. The ID is automatically generated ID for API access. Click create.

3. You will receive an API key for the new location. Copy this key and keep it safe as you will need it to add agents to the location, and you won’t be able to see it again. private location key

4. Start the Checkly agent by running the following command in a terminal, making sure to replace the `API_KEY` with the key you just received. [See more configuration options.](/docs/platform/private-locations/agent-configuration) ```bash theme={null} docker run -e API_KEY="pl_...." -d checkly/agent:latest ``` 5. Refresh the Private Locations page in the Checkly app and you will see a count of the number of running agents. agent running

6. Create a new check as you normally would in Checkly. You will now see your new private location in the list of available locations. Select it and deselect any other locations. select private location

## Using Private Locations with the CLI With the [Checkly CLI](/docs/cli/overview), you can configure any check to run on a Private Location by adding the slug name to the `privateLocations` array. For example, with an [ApiCheck construct](/docs/constructs/api-check): ```ts {title="api.check.ts"} theme={null} import { ApiCheck } from 'checkly/constructs' new ApiCheck('hello-api-1', { name: 'Hello API', activated: true, maxResponseTime: 10000, degradedResponseTime: 5000, privateLocations: ['blairs-private-network'], request: { method: 'GET', url: ' https://checklyhq.com.org/', } }) ``` ## Unavailable Private Locations If a private location has checks assigned but no Checkly agents connected in the last 10 to 20 minutes, it will be flagged as unavailable. Checkly will email account owners and admins when this happens. While a location is unavailable, no checks will be scheduled to run on it. When a location becomes available, check scheduling and execution will resume automatically. ## Using Playwright Check Suites in Private Locations We recommend to update the Checkly agent regularly. * [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) are available in private locations since Checkly Agent `6.0.3`. Please use a minimum container size of 2 CPU cores and 4 GB of RAM when running Playwright Check Suites. ## Next Steps Once your first agent is running, explore these topics to start using Private Locations in production: Detailed agent configuration options, updating your agent, runtime support, and more. Run production-grade agent deployments on a container orchestrator like Kubernetes. How to calculate the number of agents and amount of resources needed. Use the dev agent for npm packages that require compilation. # Proxy Setup Source: https://www.checklyhq.com/docs/platform/private-locations/proxy-setup Configure HTTP proxies for Checkly Private Locations in enterprise environments. Set up proxy routing for both agent management traffic and check execution. If you are operating our [Checkly Agent](/docs/platform/private-locations/overview) behind an HTTP proxy, for example in an enterprise environment, you can use an outgoing proxy for all check traffic. We recommend using the following setup. > Note you can also define a proxy for management traffic from your private location to the Checkly API where we ingest > your telemetry and management events. See [the environment variables available to the Checkly Agent container](/docs/platform/private-locations/agent-configuration) ## Setting an HTTP proxy for your Private Location We recommend storing the URL of your proxy in a [global environment variable](https://app.checklyhq.com/environment-variables) so you can easily reuse it in your Private Location configuration and checks. In the example below we store it as `PROXY_URL` private location proxy url

After this, you can reference this variable using `{{PROXY_URL}}` in the Private Location configuration. private location proxy url

## Using an HTTP proxy With the setup complete, any API check or uptime monitor which uses that Private Location will automatically inherit the proxy configuration for outgoing traffic. ## Using an HTTP proxy with Browser checks You can enable your proxy directly in your browser checks via a few extra lines of code, using the `process.env.PROXY_URL` notation and Playwright Test's `test.use()` method: ```typescript use-proxy.spec.ts theme={null} import { test } from '@playwright/test' test.use({ proxy: { server: process.env.PROXY_URL } }) test('Go to google.com', async ({ page }) => { await page.goto('https://google.com') }) ``` ```javascript use-proxy.spec.js theme={null} const { test } = require('@playwright/test') test.use({ proxy: { server: process.env.PROXY_URL } }) test('Go to google.com', async ({ page }) => { await page.goto('https://google.com') }) ``` This is all that is required for a browser check to proxy all outbound network connections via your designated HTTP proxy. Check out both the Playwright [networking docs](https://playwright.dev/docs/network#http-proxy) and the Chromium [network settings docs](https://www.chromium.org/developers/design-documents/network-settings/) for some more information. # Scaling and Redundancy Source: https://www.checklyhq.com/docs/platform/private-locations/scaling-redundancy Plan capacity, implement redundancy, and scale Checkly Private Locations for production workloads. At least two running Checkly Agents per Private Location are recommended for redundancy. Agents are stateless and can be scaled horizontally as needed. You can add additional agents to a location at any time and remove them as necessary. If one Checkly Agent fails, the other agents in the same Private Location will continue to run the checks assigned to that location. Even checks that are currently in progress on an agent that fails will be rerun by another agent after a timeout period of 300 seconds. There are two cases where checks assigned to a location could fail to run: 1. All the agents in the Private Location have failed. 2. There is insufficient agent capacity in the Private Location. **Checks will stay queued for 6 minutes** waiting for an available agent. If there are no agents available or the available agents are so busy that they cannot process all the pending checks in that timeframe, the pending checks older than 6 minutes will be lost. Agents can be scaled up and scaled out. Scaling up means adding additional CPU and memory allocations to existing agent containers. Scaling out means adding additional agent containers. ## Scale up Scaling up of individual agents is generally memory-constrained. For reference, a Browser check requires about 1.5GB of RAM and an API check requires about 150MB. The number of concurrent checks allowed to run on a single agent is controlled by the `JOB_CONCURRENCY` environment variable which defaults to 1 and can be increased to 10. We do not automatically separate browser and API checks on an agent. If you are running only Browser checks or a combination of browser and API checks in a Private Location, the formula (rounded down) is: `JOB_CONCURRENCY = Container memory allocation (GB) / 1.5` For example: if your container has 8GB of memory allocated, you should set `JOB_CONCURRENCY` to 5 (8GB / 1.5GB per check). If your Private Location is only running API checks: `JOB_CONCURRENCY = Container memory allocation (GB) / 0.15` For example: if your container has 1GB of memory allocated, you should set `JOB_CONCURRENCY` to 6 (1GB / 0.15GB per API check). With 1.5GB of memory you can set `JOB_CONCURRENCY` to the maximum value of 10. ## Scale out To determine the number of agents you need in a Private Location you first need to know the number of checks assigned to the location and their frequency. API checks can be scheduled as frequently as every 10 seconds and Browser checks as frequently as every 1 minute. Based on your configuration, you should be able to estimate how many checks will run per minute in your Private Location. API checks have a maximum running time of 30 seconds, Browser checks 4 minutes. This means that in the worst case scenario, a checkly agent with a `JOB_CONCURRENCY` of 1 can run two API checks per minute, or 1 Browser check per 4 minutes. In an average configuration this will be higher and API checks are generally faster than Browser checks. Once you have an idea of how many checks will be running and you know the per-agent `JOB_CONCURRENCY` limit based on the agent container memory allocation, you can estimate the number of agents required as `agents = ((Checks per minute * average check duration in minutes) / JOB_CONCURRENCY) + 1` * Lower `JOB_CONCURRENCY`, more agents. * More checks per minute, more agents. * Longer average check duration in minutes, more agents. I.e. if your estimate is checks run on average for 30 seconds, you plug in `0.5`. * The `+ 1` is to ensure redundant capacity in case of rolling upgrades or an agent failure. Let's look at an example: 1. `JOB_CONCURRENCY = 4` 2. I expected `35` checks per minute 3. Checks will take on average 25 seconds. This is \~`0.42` minutes. This means my estimated number of agents is `((35 * 0.42) / 4) + 1 = 4.675` or rounded up 5 agents. # Analytics API Source: https://www.checklyhq.com/docs/platform/reporting/analytics-api Query Checkly check metrics with the Analytics API, returning aggregated or raw data points grouped by location or time window for your own reporting tools. With the Analytics API, you can query all check metrics and report on Checkly data in your preferred reporting tool! The Analytics APIs allows queries across different dimensions: * **Aggregated**: get the p99, sum, standard deviation or whatever you like from DNS times for API checks, or First Contentful Paint (FCP) score for Browser checks. * **Non-aggregated**: Get the raw data points so you can aggregate yourself how you want it. Use powerful options like `quickRange` to select time window presets like `last30Days` or `lastMonth`, and group results per `location` or per `pageIndex`. For aggregated queries you can set the `aggregationInterval` to get, for instance, data aggregated per week over the last 30 days. Dive straight into the [API reference](/docs/api-reference/analytics/api-checks) or keep reading for more examples! ## Analytics API basics There are several endpoints, with one for each check or monitor type. | Endpoint | Description | API Reference | | ---------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `/v1/analytics/metrics/` | Returns all the available metrics for a specific check-type (`API`, `URL`, `TCP` or `BROWSER`) | [Learn more](/docs/api-reference/analytics/list-all-available-reporting-metrics) | | `/v1/analytics/api-checks/{checkId}` | Returns metrics for API checks | [Learn more](/docs/api-reference/analytics/api-checks) | | `/v1/analytics/browser-checks/{checkId}` | Returns metrics for Browser checks | [Learn more](/docs/api-reference/analytics/browser-checks) | | `/v1/analytics/multistep-checks/{id}` | Returns metrics for Multistep checks | [Learn more](/docs/api-reference/analytics/multistep-checks) | | `/v1/analytics/playwright-checks/{id}` | Returns metrics for Playwright check suites | [Learn more](/docs/api-reference/analytics/playwright-checks) | | `/v1/analytics/dns/{id}` | Returns metrics for DNS monitors | [Learn more](/docs/api-reference/analytics/dns-monitors) | | `/v1/analytics/heartbeat-checks/{id}` | Returns metrics for Heartbeat monitors | [Learn more](/docs/api-reference/analytics/heartbeat-checks) | | `/v1/analytics/tcp-checks/{checkId}` | Returns metrics for TCP monitors | [Learn more](/docs/api-reference/analytics/tcp-checks) | | `/v1/analytics/url-monitors/{checkId}` | Returns metrics for URL monitors | [Learn more](/docs/api-reference/analytics/url-monitors) | The specific endpoints per check type return a JSON response that includes a check's attributes, period requested, grouped `series` and a `metadata` section where you'll have additional information about the metrics to create rich reports. The example below is from a Browser check that visits three pages in a typical e-commerce flow * `/login` * `/cart` * `/cart/add` We can query the for this Browser check using a `curl` command like: ```bash theme={null} curl -x GET https://api.checklyhq.com/v1/analytics/browser-checks/79ad4fe0-589a-4924-a027-d18b12eee9cb?metrics=TTFB_avg,TTFB_p99&groupBy=pageIndex&quickRange=last7Days ``` This command add the following query params: * `metrics=TTFB_avg,TTFB_p99`: return average and p99 aggregate of the TTFB (Time to First Byte) * `groupBy=pageIndex`: group by page index, i.e. the pages visited int the Browser check. * `quickRange=last7Days`: aggregate over the last 7 days. The response looks as follows 👇 ```JSON theme={null} { "checkId": "79ad4fe0-589a-4924-a027-d18b12eee9cb", "name": "Shopping Cart - Add product", "checkType": "BROWSER", "activated": true, "muted": false, "frequency": 5, "from": "2023-01-20T12:45:57.494Z", "to": "2023-01-27T12:45:57.494Z", "tags": [ "sports" ], "series": [ { "pageIndex": 0, "data": [ { "pageUrl": "https://mycart.acme.com/login", "pageIndex": 0, "responseTime_avg": 7216.8896, "TTFB_avg": 746.9548609294588, "TTFB_p99": 1059.7 } ] }, { "pageIndex": 1, "data": [ { "pageUrl": "https://mycart.acme.com/cart", "pageIndex": 1, "responseTime_avg": 7216.8896, "TTFB_avg": 239.41101193049596, "TTFB_p99": 310.8 } ] }, { "pageIndex": 2, "data": [ { "pageUrl": "https://mycart.acme.com/cart/add", "pageIndex": 2, "responseTime_avg": 7216.8896, "TTFB_avg": 1.3732638888888888, "TTFB_p99": 0 } ] } ], "metadata": { "responseTime_avg": { "unit": "milliseconds", "label": "Response time for the full script", "aggregation": "avg" }, "TTFB_avg": { "unit": "milliseconds", "label": "Time To First Byte", "aggregation": "avg" }, "TTFB_p99": { "unit": "milliseconds", "label": "Time To First Byte", "aggregation": "p99" } }, "pagination": { "page": 1, "limit": 10 } } ``` ## Using the API with Postman Here you will see an example of a request using [Postman](https://www.postman.com/) where the `/v1/analytics/browser-check` endpoint returns aggregated data (`TTFB_p50` and `TTFB_p90`) for the last 7 days grouping by page. analytics request with postman

> You **must** add the `X-Checkly-Account` and `Authorization` headers to authenticate. ## Creating a Grafana Dashboard Following these instructions you'll learn how to use the Analytics API within a [Grafana](https://grafana.com/) dashboard: 1. Creating the Data Source First, you need to create the Data Source with the Analytics API connection. After installing the [JSON API](https://grafana.com/grafana/plugins/marcusolsson-json-datasource/) plugin you'll be able to consume any REST API and handle JSON responses. grafana data source creation

2. Configuring a component Next, you can pick a **time series** component and configure it by selecting the `Checkly Analytics API` data source and setting the required HTTP header and query parameters to fetch the JSON. You'll find detailed docs on available parameters in the [API docs](/docs/api-reference/analytics/api-checks) In the following image you'll see all required configuration for a component: * `Fields` * `Path` * `Query parameters` * `Header` * `Variables` (optional) grafana component configuration

1. Designing the dashboard You can mix multiple components to create a beautiful dashboard. The following screenshot shows a dashboard with three types of results: * aggregated * non-aggregated * and summarized grouped by location. grafana dashboard design

## Creating reports with Google Sheets You can start creating your reports by using our public Google Sheets report example template. The spreadsheet includes the `=ChecklyAPI()` function to consume the Checkly Public API just entering configuring your credentials. 1. Make a copy of our [public Google Sheets template](https://docs.google.com/spreadsheets/d/1ChcFMzuO2LOJVqu7kLGjqXf-0WmnD49_ghRLmm-OQAc/edit#gid=747858019). 2. Add your `Account Id` and `API Key` in the Apps Script code (be careful if you'll share your spreadsheet with your credentials) google sheets apps script menu

3. Navigate the `Metrics` and `Checks` pages to confirm `Checkly Public API` requests are working 4. Go to the `Analytics Report` page, pick a check, pick a metric and see if the chart shows the correct information google sheets report

You'll find dropdown cells to select the check, number of results, quick time range presets, or grouping attribute. ## Creating report using `curl` and `jq` You can even whip up a `bash` script and use `curl` and `jq` to print useful analytics to your terminal. Here is an example script that does the following: 1. It fetches your checks from the `/v1/checks` endpoint. 2. Loops over your checks and based on the check type, calls the `v1/analytics/` endpoint to gather useful data. ```bash theme={null} #! ACCOUNT_ID= TOKEN= rawChecks=$(curl -s --request GET \ --url 'https://api-test.checklyhq.com/v1/checks?limit=5&page=1' \ --header "Authorization: Bearer ${TOKEN}" \ --header "X-Checkly-Account: ${ACCOUNT_ID}") checksAndTypes=$(jq -c 'sort_by(.checkType) | [.[] | {id, checkType}]' <<< "${rawChecks}" ) printf '\n' for row in $(echo "${checksAndTypes}" | jq -c '.[]'); do checkType=$(jq -r '.checkType' <<< "${row}") checkId=$(jq -r '.id' <<< "${row}") case $checkType in BROWSER) rawMetric=$(curl -s --request GET \ --url "https://api-test.checklyhq.com/v1/analytics/browser-checks/${checkId}?metrics%5B%5D=TTFB_p99&metrics%5B%5D=FCP_p99&metrics%5B%5D=consoleErrors_sum&metrics%5B%5D=availability&quickRange=last30Days" \ --header "Authorization: Bearer ${TOKEN}" \ --header "X-Checkly-Account: ${ACCOUNT_ID}") jq -jr '.checkType, " - ", .name' <<< "${rawMetric}" printf '\n' jq -r '"availability\tTTFB\tFCP\tErrors", "--------------\t---\t---\t------", (.series[0].data[]| "\(.availability)\t\(.TTFB_p99)\t\(.FCP_p99)\t\(.consoleErrors_sum)")' <<< "${rawMetric}" | column -t -s $'\t' ;; API) rawMetric=$(curl -s --request GET \ --url "https://api-test.checklyhq.com/v1/analytics/api-checks/${checkId}?metrics%5B%5D=firstByte_p99&metrics%5B%5D=availability&metrics%5B%5D=dns_p99" \ --header "Authorization: Bearer ${TOKEN}" \ --header "X-Checkly-Account: ${ACCOUNT_ID}") jq -jr '.checkType, " - ", .name' <<< "${rawMetric}" printf '\n' jq -r '"Availability\tTTFB\tDNS", "--------------\t----\t---", (.series[0].data[]| "\(.availability)\t\(.firstByte_p99)\t\(.dns_p99)")' <<< "${rawMetric}" | column -t -s $'\t' ;; *) echo -n "unknown" ;; esac printf '%.0s\n' {1..2} done ``` Save this script in a file called `analytics.sh` and run it. You will get an output similar to: bash analytics output

# Reporting Results with Checkly Source: https://www.checklyhq.com/docs/platform/reporting/overview Analyze check performance and export monitoring data with Checkly reporting dashboard The reporting dashboard provides comprehensive insights into your monitoring performance, including response times, success ratios, and trends across all your checks. Export data for further analysis or create reports for stakeholders.

## Dashboard Overview The reporting dashboard displays your first 15 checks with key performance metrics at a glance. Use this view to: * Quickly identify underperforming checks * Monitor trends across your monitoring infrastructure * Compare performance between different check types * Spot patterns in success ratios and response times ## Filtering and Sorting Customize your view using the toolbar filters and sorting options: Select the time period for your analysis. The dashboard adjusts data granularity automatically based on your selection. Filter by: * **Check Type**: API, Browser, or Multistep checks * **Check Tags**: Filter by tags applied to individual checks * **Check Group Tags**: Filter by tags applied to check groups Sort your results by: * **Name**: Alphabetical order (ASC/DESC) * **Check Type**: Group similar check types together (ASC/DESC) ## Performance Graphs Click any check to view detailed performance graphs showing response times and success ratios over your selected time period. Reporting Dashboard Graphs

### Graph Features **Response Time Metrics**: Toggle between average, P95, and P99 response times using the metric pills beside the graph. **Success Ratio**: Track the percentage of successful check runs over time. **Dynamic Granularity**: Data grouping adjusts automatically based on your date range: | Date Range | Data Grouping | | ---------------- | ------------- | | Up to 2 weeks | Hourly | | Up to 6 months | Daily | | Up to 24 months | Weekly | | Up to 48 months | Monthly | | Beyond 48 months | Yearly | ## Data Export Export your monitoring data for external analysis, reporting, or archival purposes. ### CSV Export Process Choose the time period for your export using the date range picker. Use check type, tags, and other filters to narrow your dataset if needed. Click **Download as CSV** in the toolbar to generate your export file. ### Export Data Fields The CSV export includes the following columns: | Field | Description | | ----------------------- | ------------------------------- | | Name | Check name for identification | | Check ID | Unique identifier for the check | | Check Type | API, Browser, or Multistep | | Success Ratio | Percentage of successful runs | | Response Time (Average) | Mean response time | | Response Time (P95) | 95th percentile response time | | Response Time (P99) | 99th percentile response time | | Check Tags | Applied tags for categorization | All statistics are calculated based on your selected date range. The default range is "Last 24 Hours". ## Best Practices * **Regular exports**: Schedule periodic data exports for long-term trend analysis * **Filter strategically**: Use tags and groups to create focused reports for different teams * **Monitor P95/P99**: Don't rely only on averages - percentiles reveal user experience reality * **Compare periods**: Use date range filtering to compare performance before and after deployments # Checkly Runtimes Source: https://www.checklyhq.com/docs/platform/runtimes/overview How Checkly runtimes work. Checkly lets you use JavaScript code in your [Browser checks](/docs/detect/synthetic-monitoring/browser-checks/overview), [MultiStep Checks](/docs/detect/synthetic-monitoring/multistep-checks/overview), and optional [setup and teardown scripts](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown) for API checks. This JavaScript code executes in a runtime environment that is managed by Checkly. This environment includes a specific Node.js version, Playwright version, browser binaries, and selected npm packages. We call these managed environments **runtimes**. Playwright Check Suites use a different model. They do not use Checkly runtimes. They install your project dependencies from your `package.json` and lock file, and use `engine` to select the JavaScript engine version. ## Using runtimes You do not have to configure runtimes for most checks. 1. View all the available runtimes on the [runtimes tab in the account section](https://app.checklyhq.com/settings/account/runtimes/). See the screenshot below. 2. There is always one runtime marked as **active**. This is the runtime all checks use, unless you decide to override this at the check or group level. 3. Runtimes have a simple `YYYY.MM` name, e.g. `2026.04`. Newer dates include newer dependencies. 4. Pick the runtime you want and click "Save settings". This runtime now applies to all of your checks and any new checks you create.

Note that one runtime version, e.g. `2026.04`, holds all runtime dependencies for all check types that use Checkly runtimes. There is no separate runtime for different types of checks. ## Overriding runtimes for specific checks Before updating your Checkly account to a new runtime, you can enable a newer runtime just for one check or check group. This is a very powerful way to make sure your checks are compatible with a new runtime. You can: * Select a specific runtime in the **Browser check** editor. * Select a specific runtime in the "Setup & teardown" tab for each **API check**. * Select a specific runtime on the "Runtimes" tab for each **check group**. Playwright Check Suites do not appear in this hierarchy because they use `engine`, not `runtimeId`. > Overriding the global, account-level runtime for a few checks is **the recommended way to check for compatibility issues** before committing to a new global runtime. Note the hierarchy: the **check** runtime takes precedence over the **group** runtime, which takes precedence over the **global account** runtime. ## How we update and release new runtimes We ship new runtimes with a regular cadence. This is what you can expect from us: **1. Major & minor version upgrades of core packages and Node.js versions** We aim to release a new runtime that holds the major and minor upgrades of Playwright and Node.js **every quarter**. This will result in a new, named runtime e.g. `2023.03` available in the Checkly webapp and in our API. **2. Patch releases and security patches** We reserve the right to **update an existing runtime** with critical security patches if deemed necessary. We will only do this if there are no known backwards compatibility issues. ## Why are runtimes important? We introduced the concept of runtimes so customers can upgrade their JavaScript enhanced checks to more recent Node package versions in a controlled fashion; and roll back if anything breaks! Before, this was a more risky "big bang" migration. ## Why can't I import any NPM package or other 3rd party dependencies? Currently, Checkly is not a full-blown CI system, where a user has full control over the Node.js version and the packages installed in the `node_modules` directory. To help you navigate this tradeoff, we want to show you exactly what happens under the hood. 1. We bundle all your user code when using the [Checkly CLI](/docs/cli/overview) or GitHub Sync before storing your Check in our database. 2. Your Check is scheduled to our 20+ global locations via a message bus that has a 256kb file size limit. This is in 99% of the cases more than enough for any user code and included file dependencies you may include. However, it is not enough for potentially unlimited NPM dependencies living in `node_modules`. 3. The Checkly Runtime prepackages common 3rd party NPM modules and defines the Node.js version to avoid building, transferring and caching potentially each Check across multiple Node.js versions, greatly reducing complexity. 4. Our provided Runtime packages are tested with our sandboxing for extra runtime security. If you want to see a package included in the Checkly Runtime, [please add a post on our feedback board](https://feedback.checklyhq.com/) with a package name, and your use case. # Runtime Specification Source: https://www.checklyhq.com/docs/platform/runtimes/runtime-specification Detailed information on specific Checkly runtimes. By default, all Checkly runners (the isolated environments where checks execute) run with their time zone set to UTC, regardless of the locations used in your checks. ## Resource limitations ### Browser and Multistep Checks Browser and Multistep checks can use up to `2.7 GiB` of memory and `1678m` [milli-CPU units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu). This limit applies to all processes spawned by the check, including the browser, the test framework, and the test code itself. The limit is enforced by the runner and is not configurable. > When the memory limit is exceeded, the check will automatically fail with a relevant error message, for example: > `Your check has reached the maximum memory usage of 2.7 GiB`. ### Playwright Check Suites Playwright Check Suites run in a custom dependencies environment. Playwright Check Suites can use up to `6.8 GiB` of memory and `33000m` milli-CPU units. * **Node.js version**: v22 * **Execution limit**: 30 minutes per check run Playwright Check Suites install dependencies from your `package.json` to create the runtime environment. Learn more about [using custom dependencies](/docs/detect/synthetic-monitoring/playwright-checks/custom-dependencies). To take advantage of the available CPU cores, configure workers in your `playwright.config.ts`: ```typescript theme={null} workers: process.env.CHECKLY ? 4 : undefined, ``` ## Built-in Node.js modules The following standard Node modules are included and can be used as normal, e.g. ```ts theme={null} import * as path from 'path' ``` * assert * buffer * crypto * dns * fs (partial, we have restricted some file system operations for security reasons) * path * querystring * readline * stream * string\_decoder * timers * tls * url * util * zlib See the built-in module documentation on the official Node.js site (please see below which runtime includes what NodeJS version): * [18.x](https://nodejs.org/dist/latest-v18.x/docs/api/) * [16.x](https://nodejs.org/dist/latest-v16.x/docs/api/) ## NPM packages These are the currently available runtimes and the included external NPM dependencies. > The packages below are included for **setup and teardown scripts** as well, with the exclusion of `fs`, Playwright > and `mocha`. ### `2026.04` Node.js Version: `v24.13.1` This runtime includes Playwright v1.58.2 and Node.js v24.13.1. New dependencies are: `axios-cookiejar-support`, `csv-parse`, `imapflow`, `mysql2`, `pdf-parse`, `tough-cookie` and `winston`. Removed: `jsdom` (you can use Playwright's built-in page APIs instead). * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.58.2) 1.58.2 * [@axe-core/playwright](https://npmjs.com/package/@axe-core/playwright/v/4.11.1) 4.11.1 * [@azure/identity](https://npmjs.com/package/@azure/identity/v/4.13.0) 4.13.0 * [@azure/keyvault-secrets](https://npmjs.com/package/@azure/keyvault-secrets/v/4.10.0) 4.10.0 * [@checkly/playwright-helpers](https://npmjs.com/package/@checkly/playwright-helpers/v/1.0.4) 1.0.4 * [@clerk/testing](https://npmjs.com/package/@clerk/testing/v/1.14.2) 1.14.2 * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/10.3.0) 10.3.0 * [@google-cloud/local-auth](https://npmjs.com/package/@google-cloud/local-auth/v/3.0.1) 3.0.1 * [@kubernetes/client-node](https://npmjs.com/package/@kubernetes/client-node/v/1.4.0) 1.4.0 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.9.0) 1.9.0 * [@opentelemetry/exporter-metrics-otlp-grpc](https://npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc/v/0.212.0) 0.212.0 * [@opentelemetry/sdk-metrics](https://npmjs.com/package/@opentelemetry/sdk-metrics/v/2.5.1) 2.5.1 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/2.5.1) 2.5.1 * [@t3-oss/env-nextjs](https://npmjs.com/package/@t3-oss/env-nextjs/v/0.7.3) 0.7.3 * [@xmldom/xmldom](https://npmjs.com/package/@xmldom/xmldom/v/0.9.8) 0.9.8 * [aws4](https://npmjs.com/package/aws4/v/1.13.2) 1.13.2 * [axios](https://npmjs.com/package/axios/v/1.13.5) 1.13.5 * [axios-cookiejar-support](https://npmjs.com/package/axios-cookiejar-support/v/5.0.5) 5.0.5 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.2.0) 4.2.0 * [csv-parse](https://npmjs.com/package/csv-parse/v/6.1.0) 6.1.0 * [date-fns](https://npmjs.com/package/date-fns/v/4.1.0) 4.1.0 * [date-fns-tz](https://npmjs.com/package/date-fns-tz/v/3.2.0) 3.2.0 * [dotenv](https://npmjs.com/package/dotenv/v/17.3.1) 17.3.1 * [ethers](https://npmjs.com/package/ethers/v/6.16.0) 6.16.0 * [expect](https://npmjs.com/package/expect/v/30.2.0) 30.2.0 * [form-data](https://npmjs.com/package/form-data/v/4.0.5) 4.0.5 * [gaxios](https://npmjs.com/package/gaxios/v/7.1.3) 7.1.3 * [gmail-api-parse-message-ts](https://npmjs.com/package/gmail-api-parse-message-ts/v/2.2.33) 2.2.33 * [google-auth-library](https://npmjs.com/package/google-auth-library/v/10.6.1) 10.6.1 * [googleapis](https://npmjs.com/package/googleapis/v/171.4.0) 171.4.0 * [graphql](https://npmjs.com/package/graphql/v/16.13.0) 16.13.0 * [graphql-tag](https://npmjs.com/package/graphql-tag/v/2.12.6) 2.12.6 * [http](https://npmjs.com/package/http/v/24.13.1) 24.13.1 * [https](https://npmjs.com/package/https/v/24.13.1) 24.13.1 * [imapflow](https://npmjs.com/package/imapflow/v/1.2.10) 1.2.10 * [jose](https://npmjs.com/package/jose/v/6.1.3) 6.1.3 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/9.0.3) 9.0.3 * [lodash](https://npmjs.com/package/lodash/v/4.17.23) 4.17.23 * [long](https://npmjs.com/package/long/v/5.3.2) 5.3.2 * [mailosaur](https://npmjs.com/package/mailosaur/v/11.0.2) 11.0.2 * [moment](https://npmjs.com/package/moment/v/2.30.1) 2.30.1 * [mysql](https://npmjs.com/package/mysql/v/2.18.1) 2.18.1 * [mysql2](https://npmjs.com/package/mysql2/v/3.18.2) 3.18.2 * [nice-grpc](https://npmjs.com/package/nice-grpc/v/2.1.14) 2.1.14 * [nice-grpc-client-middleware-deadline](https://npmjs.com/package/nice-grpc-client-middleware-deadline/v/2.0.17) 2.0.17 * [nice-grpc-client-middleware-devtools](https://npmjs.com/package/nice-grpc-client-middleware-devtools/v/1.0.9) 1.0.9 * [nice-grpc-client-middleware-retry](https://npmjs.com/package/nice-grpc-client-middleware-retry/v/3.1.13) 3.1.13 * [nice-grpc-common](https://npmjs.com/package/nice-grpc-common/v/2.0.2) 2.0.2 * [nice-grpc-error-details](https://npmjs.com/package/nice-grpc-error-details/v/0.2.12) 0.2.12 * [nice-grpc-opentelemetry](https://npmjs.com/package/nice-grpc-opentelemetry/v/0.1.20) 0.1.20 * [nice-grpc-prometheus](https://npmjs.com/package/nice-grpc-prometheus/v/0.2.9) 0.2.9 * [nice-grpc-server-health](https://npmjs.com/package/nice-grpc-server-health/v/2.0.17) 2.0.17 * [nice-grpc-server-middleware-terminator](https://npmjs.com/package/nice-grpc-server-middleware-terminator/v/2.0.16) 2.0.16 * [nice-grpc-server-reflection](https://npmjs.com/package/nice-grpc-server-reflection/v/3.0.3) 3.0.3 * [nice-grpc-web](https://npmjs.com/package/nice-grpc-web/v/3.3.9) 3.3.9 * [node-pop3](https://npmjs.com/package/node-pop3/v/0.11.0) 0.11.0 * [otpauth](https://npmjs.com/package/otpauth/v/9.5.0) 9.5.0 * [pdf-parse](https://npmjs.com/package/pdf-parse/v/1.1.1) 1.1.1 * [pdf2json](https://npmjs.com/package/pdf2json/v/4.0.2) 4.0.2 * [playwright](https://npmjs.com/package/playwright/v/1.58.2) 1.58.2 * [prisma](https://npmjs.com/package/prisma/v/7.4.1) 7.4.1 * [protobufjs](https://npmjs.com/package/protobufjs/v/8.0.0) 8.0.0 * [tedious](https://npmjs.com/package/tedious/v/19.2.1) 19.2.1 * [tough-cookie](https://npmjs.com/package/tough-cookie/v/6.0.0) 6.0.0 * [twilio](https://npmjs.com/package/twilio/v/5.12.2) 5.12.2 * [uuid](https://npmjs.com/package/uuid/v/11.1.0) 11.1.0 * [winston](https://npmjs.com/package/winston/v/3.19.0) 3.19.0 * [ws](https://npmjs.com/package/ws/v/8.19.0) 8.19.0 * [xml-crypto](https://npmjs.com/package/xml-crypto/v/6.1.2) 6.1.2 * [xml-encryption](https://npmjs.com/package/xml-encryption/v/3.1.0) 3.1.0 * [zod](https://npmjs.com/package/zod/v/3.24.3) 3.24.3 ### `2025.04` Node.js Version: `v22.11.0` The main update is Playwright 1.51.1. The Node.js version is v22.11.0. * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.51.1) 1.51.1 * [@axe-core/playwright](https://npmjs.com/package/@axe-core/playwright/v/4.10.1) 4.10.1 * [@azure/identity](https://npmjs.com/package/@azure/identity/v/4.9.1) 4.9.1 * [@azure/keyvault-secrets](https://npmjs.com/package/@azure/keyvault-secrets/v/4.9.0) 4.9.0 * [@checkly/playwright-helpers](https://npmjs.com/package/@checkly/playwright-helpers/v/1.0.4) 1.0.4 * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/9.7.0) 9.7.0 * [@google-cloud/local-auth](https://npmjs.com/package/@google-cloud/local-auth/v/3.0.1) 3.0.1 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.9.0) 1.9.0 * [@opentelemetry/exporter-metrics-otlp-grpc](https://npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc/v/0.200.0) 0.200.0 * [@opentelemetry/sdk-metrics](https://npmjs.com/package/@opentelemetry/sdk-metrics/v/2.0.0) 2.0.0 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/2.0.0) 2.0.0 * [@t3-oss/env-nextjs](https://npmjs.com/package/@t3-oss/env-nextjs/v/0.13.0) 0.13.0 * [@xmldom/xmldom](https://npmjs.com/package/@xmldom/xmldom/v/0.9.8) 0.9.8 * [aws4](https://npmjs.com/package/aws4/v/1.13.2) 1.13.2 * [axios](https://npmjs.com/package/axios/v/0.28.0) 0.28.0 * [btoa](https://npmjs.com/package/btoa/v/1.2.1) 1.2.1 * [http](https://npmjs.com/package/http/v/22.11.0) 22.11.0 * [https](https://npmjs.com/package/https/v/22.11.0) 22.11.0 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.2.0) 4.2.0 * [date-fns](https://npmjs.com/package/date-fns/v/4.1.0) 4.1.0 * [date-fns-tz](https://npmjs.com/package/date-fns-tz/v/3.2.0) 3.2.0 * [dotenv](https://npmjs.com/package/dotenv/v/16.5.0) 16.5.0 * [ethers](https://npmjs.com/package/ethers/v/6.13.5) 6.13.5 * [expect](https://npmjs.com/package/expect/v/29.7.0) 29.7.0 * [form-data](https://npmjs.com/package/form-data/v/4.0.4) 4.0.4 * [gmail-api-parse-message-ts](https://npmjs.com/package/gmail-api-parse-message-ts/v/2.2.33) 2.2.33 * [google-auth-library](https://npmjs.com/package/google-auth-library/v/9.15.1) 9.15.1 * [googleapis](https://npmjs.com/package/googleapis/v/148.0.0) 148.0.0 * [graphql](https://npmjs.com/package/graphql/v/16.10.0) 16.10.0 * [graphql-tag](https://npmjs.com/package/graphql-tag/v/2.12.6) 2.12.6 * [jose](https://npmjs.com/package/jose/v/5.10.0) 5.10.0 * [jsdom](https://npmjs.com/package/jsdom/v/26.1.0) 26.1.0 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/9.0.2) 9.0.2 * [lodash](https://npmjs.com/package/lodash/v/4.17.21) 4.17.21 * [long](https://npmjs.com/package/long/v/5.3.2) 5.3.2 * [moment](https://npmjs.com/package/moment/v/2.30.1) 2.30.1 * [nice-grpc](https://npmjs.com/package/nice-grpc/v/2.1.12) 2.1.12 * [nice-grpc-client-middleware-deadline](https://npmjs.com/package/nice-grpc-client-middleware-deadline/v/2.0.15) 2.0.15 * [nice-grpc-client-middleware-devtools](https://npmjs.com/package/nice-grpc-client-middleware-devtools/v/1.0.7) 1.0.7 * [nice-grpc-client-middleware-retry](https://npmjs.com/package/nice-grpc-client-middleware-retry/v/3.1.11) 3.1.11 * [nice-grpc-common](https://npmjs.com/package/nice-grpc-common/v/2.0.2) 2.0.2 * [nice-grpc-error-details](https://npmjs.com/package/nice-grpc-error-details/v/0.2.9) 0.2.9 * [nice-grpc-opentelemetry](https://npmjs.com/package/nice-grpc-opentelemetry/v/0.1.18) 0.1.18 * [nice-grpc-prometheus](https://npmjs.com/package/nice-grpc-prometheus/v/0.2.7) 0.2.7 * [nice-grpc-server-health](https://npmjs.com/package/nice-grpc-server-health/v/2.0.14) 2.0.14 * [nice-grpc-server-middleware-terminator](https://npmjs.com/package/nice-grpc-server-middleware-terminator/v/2.0.14) 2.0.14 * [nice-grpc-server-reflection](https://npmjs.com/package/nice-grpc-server-reflection/v/2.0.14) 2.0.14 * [nice-grpc-web](https://npmjs.com/package/nice-grpc-web/v/3.3.7) 3.3.7 * [node-pop3](https://npmjs.com/package/node-pop3/v/0.9.1) 0.9.1 * [otpauth](https://npmjs.com/package/otpauth/v/9.4.0) 9.4.0 * [playwright](https://npmjs.com/package/playwright/v/1.51.1) 1.51.1 * [pdf2json](https://npmjs.com/package/pdf2json/v/3.1.4) 3.1.4 * [prisma](https://npmjs.com/package/prisma/v/6.6.0) 6.6.0 * [protobufjs](https://npmjs.com/package/protobufjs/v/7.5.0) 7.5.0 * [tedious](https://npmjs.com/package/tedious/v/18.6.1) 18.6.1 * [twilio](https://npmjs.com/package/twilio/v/5.3.0) 5.3.0 * [uuid](https://npmjs.com/package/uuid/v/11.1.0) 11.1.0 * [ws](https://npmjs.com/package/ws/v/8.18.1) 8.18.1 * [xml-crypto](https://npmjs.com/package/xml-crypto/v/6.1.1) 6.1.1 * [xml-encryption](https://npmjs.com/package/xml-encryption/v/3.1.0) 3.1.0 * [zod](https://npmjs.com/package/zod/v/3.24.3) 3.24.3 * [@clerk/testing](https://npmjs.com/package/@clerk/testing/v/1.5.1) 1.5.1 * [mailosaur](https://npmjs.com/package/mailosaur/v/8.8.1) 8.8.1 * [gaxios](https://npmjs.com/package/gaxios/v/6.7.1) 6.7.1 * [@kubernetes/client-node](https://npmjs.com/package/@kubernetes/client-node/v/1.1.2) 1.1.2 * [mysql](https://npmjs.com/package/mysql/v/2.18.1) 2.18.1 ### `2024.09` Node.js Version: `v18.20.3` The main update is Playwright 1.48.2. The Node.js version is v18.20.3. New dependencies are `@clerk/testing`, `mailosaur`, `gaxios`, `@kubernetes/client-node` and `mysql`. * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.48.2) 1.48.2 * [@axe-core/playwright](https://npmjs.com/package/@axe-core/playwright/v/4.10.0) 4.10.0 * [@azure/identity](https://npmjs.com/package/@azure/identity/v/4.4.1) 4.4.1 * [@azure/keyvault-secrets](https://npmjs.com/package/@azure/keyvault-secrets/v/4.8.0) 4.8.0 * [@checkly/playwright-helpers](https://npmjs.com/package/@checkly/playwright-helpers/v/1.0.4) 1.0.4 * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/9.0.1) 9.0.1 * [@google-cloud/local-auth](https://npmjs.com/package/@google-cloud/local-auth/v/3.0.1) 3.0.1 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.9.0) 1.9.0 * [@opentelemetry/exporter-metrics-otlp-grpc](https://npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc/v/0.53.0) 0.53.0 * [@opentelemetry/sdk-metrics](https://npmjs.com/package/@opentelemetry/sdk-metrics/v/1.26.0) 1.26.0 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/1.26.0) 1.26.0 * [@t3-oss/env-nextjs](https://npmjs.com/package/@t3-oss/env-nextjs/v/0.11.1) 0.11.1 * [@xmldom/xmldom](https://npmjs.com/package/@xmldom/xmldom/v/0.9.2) 0.9.2 * [aws4](https://npmjs.com/package/aws4/v/1.13.2) 1.13.2 * [axios](https://npmjs.com/package/axios/v/0.28.0) 0.28.0 * [btoa](https://npmjs.com/package/btoa/v/1.2.1) 1.2.1 * [http](https://npmjs.com/package/http/v/18.20.3) 18.20.3 * [https](https://npmjs.com/package/https/v/18.20.3) 18.20.3 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.2.0) 4.2.0 * [date-fns](https://npmjs.com/package/date-fns/v/3.3.1) 3.3.1 * [date-fns-tz](https://npmjs.com/package/date-fns-tz/v/3.1.3) 3.1.3 * [dotenv](https://npmjs.com/package/dotenv/v/16.4.5) 16.4.5 * [ethers](https://npmjs.com/package/ethers/v/6.13.2) 6.13.2 * [expect](https://npmjs.com/package/expect/v/29.7.0) 29.7.0 * [form-data](https://npmjs.com/package/form-data/v/4.0.4) 4.0.4 * [gmail-api-parse-message-ts](https://npmjs.com/package/gmail-api-parse-message-ts/v/2.2.33) 2.2.33 * [google-auth-library](https://npmjs.com/package/google-auth-library/v/9.14.1) 9.14.1 * [googleapis](https://npmjs.com/package/googleapis/v/144.0.0) 144.0.0 * [graphql](https://npmjs.com/package/graphql/v/16.9.0) 16.9.0 * [graphql-tag](https://npmjs.com/package/graphql-tag/v/2.12.6) 2.12.6 * [jose](https://npmjs.com/package/jose/v/5.9.2) 5.9.2 * [jsdom](https://npmjs.com/package/jsdom/v/25.0.0) 25.0.0 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/9.0.2) 9.0.2 * [lodash](https://npmjs.com/package/lodash/v/4.17.21) 4.17.21 * [long](https://npmjs.com/package/long/v/5.2.3) 5.2.3 * [moment](https://npmjs.com/package/moment/v/2.30.1) 2.30.1 * [nice-grpc](https://npmjs.com/package/nice-grpc/v/2.1.9) 2.1.9 * [nice-grpc-client-middleware-deadline](https://npmjs.com/package/nice-grpc-client-middleware-deadline/v/2.0.12) 2.0.12 * [nice-grpc-client-middleware-devtools](https://npmjs.com/package/nice-grpc-client-middleware-devtools/v/1.0.4) 1.0.4 * [nice-grpc-client-middleware-retry](https://npmjs.com/package/nice-grpc-client-middleware-retry/v/3.1.8) 3.1.8 * [nice-grpc-common](https://npmjs.com/package/nice-grpc-common/v/2.0.2) 2.0.2 * [nice-grpc-error-details](https://npmjs.com/package/nice-grpc-error-details/v/0.2.6) 0.2.6 * [nice-grpc-opentelemetry](https://npmjs.com/package/nice-grpc-opentelemetry/v/0.1.15) 0.1.15 * [nice-grpc-prometheus](https://npmjs.com/package/nice-grpc-prometheus/v/0.2.4) 0.2.4 * [nice-grpc-server-health](https://npmjs.com/package/nice-grpc-server-health/v/2.0.11) 2.0.11 * [nice-grpc-server-middleware-terminator](https://npmjs.com/package/nice-grpc-server-middleware-terminator/v/2.0.11) 2.0.11 * [nice-grpc-server-reflection](https://npmjs.com/package/nice-grpc-server-reflection/v/2.0.11) 2.0.11 * [nice-grpc-web](https://npmjs.com/package/nice-grpc-web/v/3.3.4) 3.3.4 * [otpauth](https://npmjs.com/package/otpauth/v/9.3.2) 9.3.2 * [playwright](https://npmjs.com/package/playwright/v/1.48.2) 1.48.2 * [pdf2json](https://npmjs.com/package/pdf2json/v/3.1.4) 3.1.4 * [prisma](https://npmjs.com/package/prisma/v/5.19.1) 5.19.1 * [protobufjs](https://npmjs.com/package/protobufjs/v/7.4.0) 7.4.0 * [tedious](https://npmjs.com/package/tedious/v/18.6.1) 18.6.1 * [twilio](https://npmjs.com/package/twilio/v/5.3.0) 5.3.0 * [uuid](https://npmjs.com/package/uuid/v/10.0.0) 10.0.0 * [ws](https://npmjs.com/package/ws/v/8.18.0) 8.18.0 * [xml-crypto](https://npmjs.com/package/xml-crypto/v/4.0.1) 4.0.1 * [xml-encryption](https://npmjs.com/package/xml-encryption/v/3.0.2) 3.0.2 * [zod](https://npmjs.com/package/zod/v/3.23.8) 3.23.8 * [@clerk/testing](https://npmjs.com/package/@clerk/testing/v/1.3.0) 1.3.0 * [mailosaur](https://npmjs.com/package/mailosaur/v/8.6.1) 8.6.1 * [gaxios](https://npmjs.com/package/gaxios/v/6.7.1) 6.7.1 * [@kubernetes/client-node](https://npmjs.com/package/@kubernetes/client-node/v/0.22.3) 0.22.3 * [mysql](https://npmjs.com/package/mysql/v/2.18.1) 2.18.1 ### `2024.02` Node.js Version: `v18.20.3` The main update is Playwright 1.42.1. The Node.js version is v18.20.3. New dependencies are `@opentelemetry/exporter-metrics-otlp-grpc`, `@opentelemetry/sdk-metrics`, `chai-json-schema`, `pdf2json`, `protobufjs`, `long` and the `nice-grpc` family. * [@axe-core/playwright](https://npmjs.com/package/@axe-core/playwright/v/4.8.5) 4.8.5 * [@checkly/playwright-helpers](https://npmjs.com/package/@checkly/playwright-helpers/v/1.0.2) 1.0.2 * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/8.4.1) 8.4.1 * [@google-cloud/local-auth](https://npmjs.com/package/@google-cloud/local-auth/v/3.0.1) 3.0.1 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.7.0) 1.7.0 * [@opentelemetry/exporter-metrics-otlp-grpc](https://npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc/v/0.48.0) 0.48.0 * [@opentelemetry/sdk-metrics](https://npmjs.com/package/@opentelemetry/sdk-metrics/v/1.22.0) 1.22.0 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/1.22.0) 1.22.0 * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.42.1) 1.42.1 * [@t3-oss/env-nextjs](https://npmjs.com/package/@t3-oss/env-nextjs/v/0.9.2) 0.9.2 * [@xmldom/xmldom](https://npmjs.com/package/@xmldom/xmldom/v/0.8.10) 0.8.10 * [aws4](https://npmjs.com/package/aws4/v/1.12.0) 1.12.0 * [axios](https://npmjs.com/package/axios/v/0.28.0) 0.28.0 * [btoa](https://npmjs.com/package/btoa/v/1.2.1) 1.2.1 * [http](https://npmjs.com/package/http/v/18.20.3) 18.20.3 * [https](https://npmjs.com/package/https/v/18.20.3) 18.20.3 * [chai](https://npmjs.com/package/chai/v/4.4.1) 4.4.1 * [chai-json-schema](https://npmjs.com/package/chai-json-schema/v/1.5.1) 1.5.1 * [chai-string](https://npmjs.com/package/chai-string/v/1.5.0) 1.5.0 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.2.0) 4.2.0 * [date-fns](https://npmjs.com/package/date-fns/v/3.3.1) 3.3.1 * [date-fns-tz](https://npmjs.com/package/date-fns-tz/v/3.1.3) 3.1.3 * [dotenv](https://npmjs.com/package/dotenv/v/16.4.5) 16.4.5 * [ethers](https://npmjs.com/package/ethers/v/6.11.1) 6.11.1 * [expect](https://npmjs.com/package/expect/v/29.7.0) 29.7.0 * [form-data](https://npmjs.com/package/form-data/v/4.0.0) 4.0.0 * [gmail-api-parse-message-ts](https://npmjs.com/package/gmail-api-parse-message-ts/v/2.2.32) 2.2.32 * [google-auth-library](https://npmjs.com/package/google-auth-library/v/9.6.3) 9.6.3 * [googleapis](https://npmjs.com/package/googleapis/v/133.0.0) 133.0.0 * [graphql](https://npmjs.com/package/graphql/v/16.9.0) 16.9.0 * [graphql-tag](https://npmjs.com/package/graphql-tag/v/2.12.6) 2.12.6 * [jose](https://npmjs.com/package/jose/v/5.2.2) 5.2.2 * [jsdom](https://npmjs.com/package/jsdom/v/24.0.0) 24.0.0 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/9.0.2) 9.0.2 * [lodash](https://npmjs.com/package/lodash/v/4.17.21) 4.17.21 * [long](https://npmjs.com/package/long/v/5.2.3) 5.2.3 * [moment](https://npmjs.com/package/moment/v/2.30.1) 2.30.1 * [nice-grpc](https://npmjs.com/package/nice-grpc/v/2.1.8) 2.1.8 * [nice-grpc-client-middleware-deadline](https://npmjs.com/package/nice-grpc-client-middleware-deadline/v/2.0.11) 2.0.11 * [nice-grpc-client-middleware-devtools](https://npmjs.com/package/nice-grpc-client-middleware-devtools/v/1.0.3) 1.0.3 * [nice-grpc-client-middleware-retry](https://npmjs.com/package/nice-grpc-client-middleware-retry/v/3.1.7) 3.1.7 * [nice-grpc-common](https://npmjs.com/package/nice-grpc-common/v/2.0.2) 2.0.2 * [nice-grpc-error-details](https://npmjs.com/package/nice-grpc-error-details/v/0.2.5) 0.2.5 * [nice-grpc-opentelemetry](https://npmjs.com/package/nice-grpc-opentelemetry/v/0.1.14) 0.1.14 * [nice-grpc-prometheus](https://npmjs.com/package/nice-grpc-prometheus/v/0.2.3) 0.2.3 * [nice-grpc-server-health](https://npmjs.com/package/nice-grpc-server-health/v/2.0.10) 2.0.10 * [nice-grpc-server-middleware-terminator](https://npmjs.com/package/nice-grpc-server-middleware-terminator/v/2.0.10) 2.0.10 * [nice-grpc-server-reflection](https://npmjs.com/package/nice-grpc-server-reflection/v/2.0.10) 2.0.10 * [nice-grpc-web](https://npmjs.com/package/nice-grpc-web/v/3.3.3) 3.3.3 * [otpauth](https://npmjs.com/package/otpauth/v/9.2.2) 9.2.2 * [pdf2json](https://npmjs.com/package/pdf2json/v/3.0.5) 3.0.5 * [playwright](https://npmjs.com/package/playwright/v/1.42.1) 1.42.1 * [prisma](https://npmjs.com/package/prisma/v/5.10.2) 5.10.2 * [protobufjs](https://npmjs.com/package/protobufjs/v/7.2.6) 7.2.6 * [protobufjs/minimal](https://npmjs.com/package/protobufjs/minimal/v/7.2.6) 7.2.6 * [protobufjs/light](https://npmjs.com/package/protobufjs/light/v/7.2.6) 7.2.6 * [twilio](https://npmjs.com/package/twilio/v/4.23.0) 4.23.0 * [uuid](https://npmjs.com/package/uuid/v/9.0.1) 9.0.1 * [ws](https://npmjs.com/package/ws/v/8.16.0) 8.16.0 * [xml-crypto](https://npmjs.com/package/xml-crypto/v/4.0.1) 4.0.1 * [xml-encryption](https://npmjs.com/package/xml-encryption/v/3.0.2) 3.0.2 * [zod](https://npmjs.com/package/zod/v/3.22.4) 3.22.4 * [@azure/identity](https://npmjs.com/package/@azure/identity/v/4.3.0) 4.3.0 * [@azure/keyvault-secrets](https://npmjs.com/package/@azure/keyvault-secrets/v/4.8.0) 4.8.0 * [tedious](https://npmjs.com/package/tedious/v/18.2.0) 18.2.0 * [allure-playwright](https://npmjs.com/package/allure-playwright/v/latest) latest * [@dvag/playwright-utils](https://npmjs.com/package/@dvag/playwright-utils/v/latest) latest ### `2023.09` Node.js Version: `v18.20.3` The main updates are Playwright 1.38.1 and the addition of ethers 6.7.1, prisma 5.1.1, zod 3.22.2, @t3-oss/env-nextjs 0.6.1 and @xmldom/xmldom 0.8.10. The Node.js version is v18.20.3. * [@checkly/playwright-helpers](https://npmjs.com/package/@checkly/playwright-helpers/v/1.0.2) 1.0.2 * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/8.0.2) 8.0.2 * [@google-cloud/local-auth](https://npmjs.com/package/@google-cloud/local-auth/v/3.0.0) 3.0.0 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.4.1) 1.4.1 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/1.15.2) 1.15.2 * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.38.1) 1.38.1 * [@t3-oss/env-nextjs](https://npmjs.com/package/@t3-oss/env-nextjs/v/0.6.1) 0.6.1 * [@xmldom/xmldom](https://npmjs.com/package/@xmldom/xmldom/v/0.8.10) 0.8.10 * [aws4](https://npmjs.com/package/aws4/v/1.12.0) 1.12.0 * [axios](https://npmjs.com/package/axios/v/0.27.2) 0.27.2 * [btoa](https://npmjs.com/package/btoa/v/1.2.1) 1.2.1 * [http](https://npmjs.com/package/http/v/18.20.3) 18.20.3 * [https](https://npmjs.com/package/https/v/18.20.3) 18.20.3 * [chai](https://npmjs.com/package/chai/v/4.3.7) 4.3.7 * [chai-string](https://npmjs.com/package/chai-string/v/1.5.0) 1.5.0 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.1.1) 4.1.1 * [date-fns](https://npmjs.com/package/date-fns/v/2.30.0) 2.30.0 * [date-fns-tz](https://npmjs.com/package/date-fns-tz/v/2.0.0) 2.0.0 * [dotenv](https://npmjs.com/package/dotenv/v/16.3.1) 16.3.1 * [ethers](https://npmjs.com/package/ethers/v/6.7.1) 6.7.1 * [expect](https://npmjs.com/package/expect/v/29.6.2) 29.6.2 * [form-data](https://npmjs.com/package/form-data/v/4.0.0) 4.0.0 * [gmail-api-parse-message-ts](https://npmjs.com/package/gmail-api-parse-message-ts/v/2.2.32) 2.2.32 * [google-auth-library](https://npmjs.com/package/google-auth-library/v/9.0.0) 9.0.0 * [googleapis](https://npmjs.com/package/googleapis/v/126.0.0) 126.0.0 * [jose](https://npmjs.com/package/jose/v/4.14.4) 4.14.4 * [jsdom](https://npmjs.com/package/jsdom/v/22.1.0) 22.1.0 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/9.0.1) 9.0.1 * [lodash](https://npmjs.com/package/lodash/v/4.17.21) 4.17.21 * [moment](https://npmjs.com/package/moment/v/2.29.4) 2.29.4 * [otpauth](https://npmjs.com/package/otpauth/v/9.1.4) 9.1.4 * [playwright](https://npmjs.com/package/playwright/v/1.38.1) 1.38.1 * [prisma](https://npmjs.com/package/prisma/v/5.1.1) 5.1.1 * [twilio](https://npmjs.com/package/twilio/v/4.15.0) 4.15.0 * [uuid](https://npmjs.com/package/uuid/v/9.0.0) 9.0.0 * [ws](https://npmjs.com/package/ws/v/8.13.0) 8.13.0 * [xml-crypto](https://npmjs.com/package/xml-crypto/v/4.1.0) 4.1.0 * [xml-encryption](https://npmjs.com/package/xml-encryption/v/3.0.2) 3.0.2 * [zod](https://npmjs.com/package/zod/v/3.22.2) 3.22.2 ### `2023.02` Node.js Version: `v16.20.2` The main updates are Playwright 1.32.1, faker 7.6.0 and the addition of date-fns 2.29.3 and ws 8.13.0. We are dropping support for Mocha. * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/7.6.0) 7.6.0 * [@google-cloud/local-auth](https://npmjs.com/package/@google-cloud/local-auth/v/2.1.1) 2.1.1 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.0.4) 1.0.4 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/1.0.1) 1.0.1 * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.32.3) 1.32.3 * [@xmldom/xmldom](https://npmjs.com/package/@xmldom/xmldom/v/0.8.10) 0.8.10 * [aws4](https://npmjs.com/package/aws4/v/1.11.0) 1.11.0 * [axios](https://npmjs.com/package/axios/v/0.27.2) 0.27.2 * [btoa](https://npmjs.com/package/btoa/v/1.2.1) 1.2.1 * [http](https://npmjs.com/package/http/v/16.20.2) 16.20.2 * [https](https://npmjs.com/package/https/v/16.20.2) 16.20.2 * [chai](https://npmjs.com/package/chai/v/4.3.7) 4.3.7 * [chai-string](https://npmjs.com/package/chai-string/v/1.5.0) 1.5.0 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.1.1) 4.1.1 * [date-fns](https://npmjs.com/package/date-fns/v/2.29.3) 2.29.3 * [date-fns-tz](https://npmjs.com/package/date-fns-tz/v/2.0.0) 2.0.0 * [expect](https://npmjs.com/package/expect/v/29.3.1) 29.3.1 * [form-data](https://npmjs.com/package/form-data/v/4.0.0) 4.0.0 * [gmail-api-parse-message-ts](https://npmjs.com/package/gmail-api-parse-message-ts/v/2.2.32) 2.2.32 * [google-auth-library](https://npmjs.com/package/google-auth-library/v/8.8.0) 8.8.0 * [googleapis](https://npmjs.com/package/googleapis/v/118.0.0) 118.0.0 * [jose](https://npmjs.com/package/jose/v/4.14.1) 4.14.1 * [jsdom](https://npmjs.com/package/jsdom/v/21.1.2) 21.1.2 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/9.0.0) 9.0.0 * [lodash](https://npmjs.com/package/lodash/v/4.17.21) 4.17.21 * [moment](https://npmjs.com/package/moment/v/2.29.4) 2.29.4 * [otpauth](https://npmjs.com/package/otpauth/v/9.0.2) 9.0.2 * [twilio](https://npmjs.com/package/twilio/v/4.11.1) 4.11.1 * [playwright](https://npmjs.com/package/playwright/v/1.32.3) 1.32.3 * [typescript](https://npmjs.com/package/typescript/v/4.8.4) 4.8.4 * [uuid](https://npmjs.com/package/uuid/v/9.0.0) 9.0.0 * [ws](https://npmjs.com/package/ws/v/8.13.0) 8.13.0 * [xml-crypto](https://npmjs.com/package/xml-crypto/v/4.0.1) 4.0.1 * [xml-encryption](https://npmjs.com/package/xml-encryption/v/3.0.2) 3.0.2 ### `2022.10` Node.js Version: `v16.20.2` The main updates are Playwright 1.28.0, Node.js v16.20.2 and Typescript support. We are dropping support for Puppeteer. * [@faker-js/faker](https://npmjs.com/package/@faker-js/faker/v/5.5.3) 5.5.3 * [@opentelemetry/api](https://npmjs.com/package/@opentelemetry/api/v/1.0.4) 1.0.4 * [@opentelemetry/sdk-trace-base](https://npmjs.com/package/@opentelemetry/sdk-trace-base/v/1.0.1) 1.0.1 * [@playwright/test](https://npmjs.com/package/@playwright/test/v/1.28.0) 1.28.0 * [aws4](https://npmjs.com/package/aws4/v/1.11.0) 1.11.0 * [axios](https://npmjs.com/package/axios/v/0.27.2) 0.27.2 * [btoa](https://npmjs.com/package/btoa/v/1.2.1) 1.2.1 * [http](https://npmjs.com/package/http/v/16.20.2) 16.20.2 * [https](https://npmjs.com/package/https/v/16.20.2) 16.20.2 * [chai](https://npmjs.com/package/chai/v/4.3.7) 4.3.7 * [chai-string](https://npmjs.com/package/chai-string/v/1.5.0) 1.5.0 * [crypto-js](https://npmjs.com/package/crypto-js/v/4.1.1) 4.1.1 * [expect](https://npmjs.com/package/expect/v/29.3.1) 29.3.1 * [form-data](https://npmjs.com/package/form-data/v/4.0.0) 4.0.0 * [jsonwebtoken](https://npmjs.com/package/jsonwebtoken/v/8.5.1) 8.5.1 * [lodash](https://npmjs.com/package/lodash/v/4.17.21) 4.17.21 * [mocha](https://npmjs.com/package/mocha/v/10.1.0) 10.1.0 * [moment](https://npmjs.com/package/moment/v/2.29.4) 2.29.4 * [otpauth](https://npmjs.com/package/otpauth/v/9.0.2) 9.0.2 * [playwright](https://npmjs.com/package/playwright/v/1.28.0) 1.28.0 * [typescript](https://npmjs.com/package/typescript/v/4.8.4) 4.8.4 * [uuid](https://npmjs.com/package/uuid/v/9.0.0) 9.0.0 # Storing Secrets Source: https://www.checklyhq.com/docs/platform/secrets Storing sensitive data in Checkly Secrets in Checkly are a specialized form of data designed specifically for sensitive information that needs to be kept secure while remaining accessible to your monitoring. Secrets allow you to store sensitive data for use in checks. Once saved secrets are never shown in the UI or in logs. The secret value cannot be accessed via the CLI or API. Secrets handle the truly sensitive elements—API keys, passwords, authentication tokens, and any other information that could compromise security if exposed. Both variables and secrets are encrypted at rest and in flight. However, Secrets go further by ensuring that once stored, their values become completely invisible to users, appearing only as masked values in interfaces while remaining fully functional in your monitoring code. If you manage many credentials with Monitoring as Code, learn how to [load them from an external secrets manager and deploy them at group or check scope](/docs/platform/manage-secrets-at-scale). ## Secrets in Practice Secrets integrate seamlessly into your monitoring workflow while maintaining security. You reference them in your code using the same syntax as regular environment variables, but they remain protected throughout the entire execution pipeline. This means you can write monitoring logic that authenticates with services, accesses protected endpoints, and performs realistic user scenarios without compromising security. Use secrets in your scripts using standard Node.js syntax: `process.env.MY_SECRET` or by using `{{handlebars}}` syntax in applicable API check fields. ## Real-World Secrets Examples Here's how you might use Secrets in real monitoring scenarios: ### API Authentication Example: ```typescript theme={null} import { ApiCheck, AssertionBuilder } from 'checkly/constructs' new ApiCheck('authenticated-api-check', { name: 'User Profile API', request: { method: 'GET', url: 'https://api.example.com/user/profile', headers: [ // Secret API key - never visible in logs or UI { key: 'Authorization', value: 'Bearer {{API_SECRET_KEY}}' }, { key: 'Content-Type', value: 'application/json' } ], assertions: [ AssertionBuilder.statusCode().equals(200), AssertionBuilder.jsonBody('user.email').isNotNull() ] } }) ``` ### Browser Login Flow Example: ```typescript theme={null} import { test, expect } from '@playwright/test' test('Login flow', async ({ page }) => { await page.goto('https://app.example.com/login') // Using secrets for login credentials await page.fill('#email', process.env.TEST_USER_EMAIL!) await page.fill('#password', process.env.TEST_USER_PASSWORD!) await page.click('#login-button') // Verify successful login await expect(page.locator('.dashboard')).toBeVisible() }) ``` ### Database Connection Example: ```typescript theme={null} import { ApiCheck } from 'checkly/constructs' new ApiCheck('database-health-check', { name: 'Database Health Check', request: { method: 'POST', url: 'https://api.example.com/health/database', headers: [ { key: 'Authorization', value: 'Bearer {{DB_MONITORING_TOKEN}}' } ], body: JSON.stringify({ // Secret database connection string connectionString: '{{DATABASE_CONNECTION_STRING}}' }) } }) ``` ### Managing Secrets via CLI: ```bash theme={null} # Create a secret npx checkly env add API_SECRET_KEY your_secret_api_key_here --secret # Create a secret for database credentials npx checkly env add DATABASE_PASSWORD super_secret_password --secret # List all variables (secrets show only key names) npx checkly env ls # Update a secret (must include --secret flag) npx checkly env update API_SECRET_KEY new_secret_value --secret ``` ### Payment Processing Example: ```typescript theme={null} import { test } from '@playwright/test' test('Checkout flow with test payment', async ({ page }) => { await page.goto('https://shop.example.com/checkout') // Fill checkout form await page.fill('#card-number', process.env.TEST_CARD_NUMBER!) await page.fill('#card-expiry', process.env.TEST_CARD_EXPIRY!) await page.fill('#card-cvc', process.env.TEST_CARD_CVC!) // Process payment await page.click('#pay-button') // Verify success await expect(page.locator('.payment-success')).toBeVisible() }) ``` ## Secrets as Security Foundation Secrets represent the foundation of secure monitoring practices. They enable you to test and monitor authenticated workflows, private APIs, and sensitive user journeys while maintaining the security posture that modern applications require. By centralizing secret management within your monitoring platform, you eliminate the risk of credentials being scattered across scripts, configuration files, or team communications. The power of Secrets lies in making security invisible to your monitoring logic—your Checks work exactly the same whether they're using public endpoints or accessing the most sensitive parts of your application, but the security model ensures that sensitive information never leaves the protected environment. ## Alternative: Dynamic Secret Detection For browser checks and multistep checks, you can also use dynamic secret detection. This approach allows you to retrieve secrets at runtime (from external vaults like Azure Key Vault, AWS Secrets Manager, etc.) and have them automatically scrubbed from logs and traces. ```javascript theme={null} // Example: Retrieve from external vault at runtime process.env.CHECKLY_SECRET_API_KEY = await getFromAzureKeyVault('api-key') process.env.CHECKLY_SECRET_TOKEN = await fetchFromAWS('auth-token') ``` This is particularly useful for organizations that prefer not to store secrets in Checkly and want to maintain their existing secret management infrastructure. [Learn more about dynamic secret scrubbing](/docs/platform/dynamic-secret-scrubbing). ## Using secrets in alert channels Secrets also work in alert-channel configurations — webhook URLs, API keys, PagerDuty service keys, and any other literal-secret field accept the same `{{NAME}}` reference syntax. See [Alert channels](/docs/communicate/alerts/channels) for details and the inline promote-to-secret affordance. ### Security and Usability Balance Use dedicated test users, test cards etc. These test users should have minimal privileges in your app. Do not use your admin or root user. Make sure you can easily disable or block these users without recourse. Keep secrets separate from your browser scripts and store them as environment secrets in Checkly. This way you can reuse secrets in multiple scripts and rotate them as needed. # Environment variables and secrets Source: https://www.checklyhq.com/docs/platform/variables Store and manage common configuration values across your checks Your checks might share the same configuration data, like an authentication token, a user name, or even a specific part of the URL. You can use variables and secrets to 'DRY' up your checks, store these variables in one place, and keep sensitive data secure. ## Overview There are two ways to store configuration information in Checkly: Variables and secrets. Both variables and secrets are encrypted at rest and in flight. * **Variables** are used to store non-sensitive information. Variables are shown in plaintext when being edited, on the check result page and in logs. Variables can be accessed via the CLI and API. * **Secrets** allow you to store sensitive data for use in checks. Once saved, secrets are never shown in the UI or in logs and cannot be accessed via the CLI or API. Secrets are fully supported starting with runtime version 2024.09 and later. For [Private Locations](/docs/platform/private-locations/overview/), secrets are available in agent version `3.3.4` and later, and for the [CLI](/docs/cli/overview/), in version `4.9.0` and later. To ensure the integrity of Playwright artifacts (traces, videos and screenshots), the following are not scrubbed, even when saved as secrets: The characters `/` and `*` and the full or partial match of `/artifact/`, `https://`, `http://`, `*********`, and `123`. Values of the keys `sha1`, `_sha1`, `pageref`, `downloadsPath`, `tracesDir`, `pageId` and any string that ends with `sha1` will not be scrubbed from the Playwright trace, but will be scrubbed from the general check result. Numbers are not scrubbed from the Playwright trace, but from the general check result. This page provides a general overview of environment variables. For information specific to the CLI, refer to our [Checkly CLI documentation](/docs/cli/environment-variables). From here on, in this document, we refer to both variables and secrets as 'variables' for ease of reading, unless explicitly mentioned. ## Managing variables You can create environment variables and secrets at three hierarchical levels: Variables defined at the check level are only available to that specific check. Use these for check-specific configuration or to override group/global variables. **Supported by:** API (only via the CLI), Browser, Multistep & Playwright checks Check-level variables are shown in the **Variables** tab on the check edit page. check environment variables

Group variables are only accessible in the context of a group, which includes the checks within the group and their configuration. Group-level variables are shown the **Variables** tab in a check group. set group environment variable

Variables defined at the global level are available throughout Checkly, including in checks, alert channels, and global configuration options. Global variables are shown in the [Environment variables](https://app.checklyhq.com/environment-variables) section on the left-side menu. set global environment variable

Store variables at the global level whenever possible to follow the DRY (Don't Repeat Yourself) principle. By default, all variables are stored as string values. When using variables, you can click the lock icon to hide the value. Any data you "lock" is encrypted at rest and in flight on our back end and is only decrypted when needed. Locked environment variables can only be accessed by team members with [Read & Write, Admin, or Owner roles](/docs/admin/team-management/overview/). Secrets are never visible for any user and are always encrypted. ## Variable hierarchy As checks are scheduled, Checkly merges the check, group, and global environment variables into one data set and exposes them to the runtime environment. During merging, variables at more specific levels override those at broader levels. Or, in other words: **check** variables override **group** variables override **global** variables. You can make use of this by providing a default value for a specific variable at the global or group level, but allow that variable to be overridden at the check level. ## Accessing variables How variables are accessed depends on where you're accessing them from: * In [Browser](/docs/detect/synthetic-monitoring/browser-checks/mac-structure#environment-variables), [Multistep](/docs/detect/synthetic-monitoring/multistep-checks/overview#built-in-runtime-variables), and [API Check setup/teardown scripts](/docs/detect/synthetic-monitoring/api-checks/setup-and-teardown#built-in-variables), use the standard Node.js `process.env.VARIABLE_NAME` notation. * In [API Check requests](/docs/detect/synthetic-monitoring/api-checks/configuration#accessing-environment-variables), use Handlebars/Moustache templating delimiters, i.e. `{{VARIABLE_NAME}}` * In [webhook alert channels](/docs/integrations/alerts/webhooks#template-variables), also use the Handlebar/Moustache format, i.e. `{{VARIABLE_NAME}}` Handlebar (double brackets) variables will be URI encoded. To avoid encoding, you can access your environment variables with triple brackets, i.e. `{{{VARIABLE_NAME}}}`. ## Playwright Check Suite Variables Checkly provides a set of [built-in environment variables](/docs/detect/synthetic-monitoring/playwright-checks/environment-variables) that you can use in your [Playwright Check Suites](/docs/detect/synthetic-monitoring/playwright-checks/overview) to distinguish between Checkly executions and local runs. # Checkly Quickstart Source: https://www.checklyhq.com/docs/quickstart Add synthetic monitoring to your existing project in minutes. Let your AI agent set up Checkly, or follow the steps yourself. Add synthetic monitoring to a project you already have, in any language or framework, in minutes. You'll end up with checks running around the clock from global locations, alerting you the moment your site or API breaks. The fastest path is to let your AI coding agent set it up: `npx checkly init` installs the [Checkly CLI](/docs/cli/overview) and [agent skills](/docs/ai/skills), and from there your agent can generate, test, and deploy your monitoring as [Monitoring as Code](/docs/concepts/monitoring-as-code) that lives alongside your application code. The steps below explain each phase so you understand what the agent does, and you can run them yourself if you prefer. *The recommended flow, end to end. Each phase is explained below.* ## Get started with Monitoring as Code * [Node.js](https://nodejs.org) `20.19+` or `22.12+`, and npm * A terminal * (Optional) the URL of an app or API you want to monitor ### Let your AI agent set it up (recommended) Run `npx checkly init` and answer a few prompts to let your agent take over: ```text Terminal theme={null} npx checkly init ✔ Do you want your AI agent to set up Checkly? … yes ✔ Install the Checkly skill for your AI coding agent? … yes ✔ Which AI coding agent do you use? › Claude ✔ Run npm install to install them now? … yes ... ✔ Copy prompt to clipboard? … yes You're ready to go! ``` `checkly init` installs the [Checkly CLI](/docs/cli/overview) and [agent skills](/docs/ai/skills), creates a `checkly.config.ts`, installs dependencies, and **copies a starter prompt tailored to your project to your clipboard**. Paste it into your AI agent and follow the instructions. From here, what the agent does depends on your project. It might: * Scan your project and discover monitoring targets * Create Checkly monitors in code * Test them with `npx checkly test` * Set up alert channels * Reuse your existing Playwright tests * Deploy your monitoring with `npx checkly deploy` after your confirmation It asks for input along the way. Open [your Checkly dashboard](https://app.checklyhq.com) to see your monitoring live. To see what each command does, or to run them yourself, follow the [steps below](#set-it-up-yourself). ### Set it up yourself The steps below use the Checkly CLI directly so you can build and deploy your monitoring by hand. Run the init command in your project directory and decline the agent setup. It scaffolds a working example you can edit. ```text Terminal theme={null} npx checkly init ✔ Do you want your AI agent to set up Checkly? … no ✔ Run npm install to install them now? … yes ✔ Add some demo checks to get started? … yes ✔ Created __checks__/ with example checks ... All done! ``` This scaffolds a `checkly.config.ts` file and a `__checks__` folder containing a few starter checks: ```bash Project structure theme={null} |- checkly.config.ts |__checks__ |- api.check.ts |- heartbeat.check.ts |- homepage.spec.ts |- url.check.ts ``` The `checkly.config.ts` file holds your project-wide settings, such as the default run frequency, locations, and which files contain your checks. Log in to your Checkly account, or create a free one, right from the terminal. ```bash Terminal theme={null} npx checkly login ``` This opens your browser to authenticate. Once you're logged in, you can run and deploy checks from your machine. Run `npx checkly whoami` at any time to confirm which account you're connected to. Checks are code. The scaffolded `__checks__/homepage.spec.ts` is a [Browser Check](/docs/detect/synthetic-monitoring/browser-checks/quickstart): a standard `@playwright/test` file that Checkly runs as a monitor. Edit it to load a page and assert on what matters: ```ts homepage.spec.ts theme={null} import { test, expect } from '@playwright/test' test('homepage loads', async ({ page }) => { const response = await page.goto('https://www.checklyhq.com') expect(response?.status()).toBeLessThan(400) await expect(page).toHaveTitle(/Checkly/) await page.screenshot({ path: 'homepage.jpg' }) }) ``` ```js homepage.spec.js theme={null} const { test, expect } = require('@playwright/test') test('homepage loads', async ({ page }) => { const response = await page.goto('https://www.checklyhq.com') expect(response?.status()).toBeLessThan(400) await expect(page).toHaveTitle(/Checkly/) await page.screenshot({ path: 'homepage.jpg' }) }) ``` Checkly turns every `*.spec.ts` file into a Browser Check through the `browserChecks.testMatch` pattern in your `checkly.config.ts`. To monitor an endpoint instead, edit `__checks__/api.check.ts`, which defines an [API Check](/docs/detect/synthetic-monitoring/api-checks/quickstart). For every construct and option, see the [Constructs reference](/docs/constructs/overview). Dry-run all your checks against Checkly's global infrastructure before you deploy anything: ```bash Terminal theme={null} npx checkly test ``` You'll see the results in your terminal: ``` Running 2 checks in eu-west-1. __checks__/homepage.spec.ts ✔ homepage loads (24s) __checks__/api.check.ts ✔ Hello API (222ms) 2 passed, 2 total ``` Runs are recorded as [test sessions](/docs/detect/testing/overview) by default, with full logs, traces, and videos you can review in the [Checkly web app](https://app.checklyhq.com). Deploy your checks and related resources to Checkly. From now on, they run around the clock from Checkly's global locations: ```bash Terminal theme={null} npx checkly deploy ``` Open [your Checkly dashboard](https://app.checklyhq.com) and you'll see your checks, ready to start monitoring. Add an [alert channel](/docs/communicate/alerts/overview) so you're notified when a check fails. Define the channels you want in a new `__checks__/alert-channels.ts` file, then attach them to your checks through the `alertChannels` defaults in your config. ```ts alert-channels.ts theme={null} import { EmailAlertChannel } from 'checkly/constructs' export const emailChannel = new EmailAlertChannel('email-alert-channel', { address: 'alerts@example.com', }) ``` ```ts alert-channels.ts theme={null} import { PhoneCallAlertChannel } from 'checkly/constructs' export const phoneChannel = new PhoneCallAlertChannel('phone-alert-channel', { name: 'On-call engineer', phoneNumber: '+1234567890', }) ``` Import the channels you defined and set them as defaults in your `checkly.config.ts` so every check, including your Browser Check, uses them. This example assumes you defined both, so import only the channels you created: ```ts checkly.config.ts theme={null} import { defineConfig } from 'checkly' import { emailChannel, phoneChannel } from './__checks__/alert-channels' export default defineConfig({ projectName: 'My Project', logicalId: 'my-project', checks: { // ...your existing check defaults alertChannels: [emailChannel, phoneChannel], }, }) ``` You can also attach `alertChannels` to an individual check or a [check group](/docs/constructs/check-group-v2) for finer control. Run `npx checkly deploy` again to apply your alerting, and see the [alert channels overview](/docs/communicate/alerts/channels) for every supported channel. When you're done experimenting, tear down everything this project created with `npx checkly destroy`. ## Go deeper Every check, monitor, and alert channel you can define as code. All `npx checkly` commands and options. Monitor realistic user flows with Playwright. Monitor your backend services and endpoints. Configure who gets notified, and how. Run and deploy your checks from your pipeline. # Rocky AI Root Cause Analysis Source: https://www.checklyhq.com/docs/resolve/ai-root-cause-analysis/overview Automatically perform root cause and user impact analysis with Rocky AI ## What is Rocky AI Root Cause Analysis? Rocky AI — Checkly's Agent — can automatically analyze, categorize and summarize any failure or error from one of Checkly's synthetic checks and / or monitoring checks. Light mode Rocky AI Root Cause Analysis

Using any available artifacts like error message, user code, check description, logs, metrics, trace routes, Open Telemetry traces, packet captures, Playwright traces, screenshots and other evidence, it will quickly assess what the impact is of a failure and what the underlying (possible) root cause is. It will also provide a suggested code fix for any check based on (Playwright) code. ## Getting Started with Rocky AI Root Cause Analysis To use Rocky AI Root Cause Analysis, make sure [Rocky AI is enabled on its settings page](https://app.checklyhq.com/accounts/rocky-ai). Rocky AI Root Cause Analysis is available for all synthetic and monitor check types in both testing and monitoring use cases. ### Automatic analysis When Rocky AI is enabled, it automatically generates a root cause analysis for every new error group. This is **on by default** — no manual action is needed for new errors to be analyzed. Each automatic analysis counts as **one AI invocation** against your [plan's invocation quota](/docs/ai/rocky-ai#consumption-and-invocations). If the same analysis is sent to multiple alert channels, it still counts as a single invocation. Note that this does not mean that if a check starts failing at midnight and you only resolve it in the morning you will blow through your invocation credits: **Rocky AI only triggers an automatic analysis the first time it sees a new error**, which results in an error group. Any subsequent, similar errors are just attributed to that error group. You can disable automatic analysis in the [Rocky AI settings](https://app.checklyhq.com/accounts/rocky-ai) if you prefer to only run analyses manually. When automatic analysis is enabled, AI-generated fields like root cause and user impact are included in your alert notifications. You can use these fields in custom webhook payloads — see [Using AI analysis data in webhooks](/docs/integrations/alerts/webhooks#using-ai-analysis-data-in-alerts). ### On-demand analysis #### Via the UI Click the "Analyze root cause" button on any failing check result, test session result and / or error group. Light mode Rocky AI Root Cause Analysis

#### Via the CLI Use [`checkly rca run`](/docs/cli/checkly-rca/#checkly-rca-run) to trigger a new analysis for an error group. ```bash theme={null} $ npx checkly rca run --error-group 714... --watch Root cause analysis triggered. RCA ID: 8c3... ROOT CAUSE ANALYSIS Classification: CONFIGURATION_ERROR Root cause: The check is calling a ... ``` Retrieve an existing analysis by error group with [`checkly checks get --error-group`](/docs/cli/checkly-checks/#checkly-checks-get) or by RCA ID with [`checkly rca get`](/docs/cli/checkly-rca/#checkly-rca-get). ```bash checkly checks get theme={null} $ npx checkly checks get d3c... --error-group 714... ERROR GROUP HTTP 404 Not Found First seen: 2026-04-21 11:39:15 UTC Last seen: 2026-04-21 11:58:15 UTC Error group: 714... ROOT CAUSE ANALYSIS Classification: CONFIGURATION_ERROR Root cause: The check is calling a ... ``` ```bash checkly rca get theme={null} $ npx checkly rca get 8c3... ROOT CAUSE ANALYSIS Classification: CONFIGURATION_ERROR Root cause: The check is calling a ... ``` ## Key Capabilities * **Categorize**: Rocky AI determines if an error is of a specific category. For example: is it a typo in your code or is it an infrastructure error? * **Root cause**: Rocky AI analyzes the failure and provides you with a root cause analysis, helping you quickly remediate the issue. It uses any and all available artifacts to come to this analysis and list these artifacts as "evidence". * **End-user impact**: Rocky AI analyzes the failure and provides you with a list of impacted user groups and features. The text used is generated by AI and can be enriched with organisation-specific domain knowledge using the **Tag description editor** the [AI settings screen](https://app.checklyhq.com/settings/account/ai). * **Step summary**: Where applicable, Rocky AI summarizes the steps in the check script, making it easier to understand what the check is doing and where it failed. This typically works for any Playwright based check. Light mode Rocky AI Root Cause Analysis

## Benefits * **Faster Triaging**: Determine if it is a typo in a script or an infrastructure error * **Reduced Manual Work**: Automate routine analysis tasks * **Better Understanding**: Gain deeper insights into system behavior * **Proactive Management**: Identify issues before they become problems For more information on AI, security and data privacy [see the general Rocky AI page](/docs/ai/rocky-ai) # Checkly Resolve Source: https://www.checklyhq.com/docs/resolve/overview Accelerate incident resolution with distributed tracing and AI-powered root cause analysis. **Checkly Resolve** provides advanced debugging and incident resolution capabilities through distributed tracing and AI-powered analysis. When issues are detected, Resolve helps you quickly understand the root cause and get back to normal operations. Analyze request flows with distributed tracing to identify performance bottlenecks and failures Let Rocky AI analyze errors, metrics, traces and logs and determine impact and root cause ## Key Benefits * **Fast Root Cause Analysis** — Identify the exact source of issues with detailed request tracing across your entire application stack. * **AI-Powered Insights** — Get intelligent analysis and recommendations from Rocky AI to resolve incidents faster than ever. * **Full Request Visibility** — See complete request flows from frontend to backend, including database queries and external API calls. * **Performance Optimization** — Identify slow operations and bottlenecks to proactively improve application performance. ## Getting Started Ready to accelerate your incident resolution? Choose the approach that best fits your current needs: Begin collecting distributed traces for your applications Experience AI-powered root cause analysis and insights Add OpenTelemetry instrumentation to your codebase Connect traces with your existing observability platform # Exporting Traces to Dash0 Source: https://www.checklyhq.com/docs/resolve/traces/export/dash0 Learn how to export traces from Checkly to Dash0. 1. Create an API key in the **Settings** > **Auth Tokens** section. Generate an auth token

2. Grab the right endpoint URL for your Dash0 organization from the **Settings** > **Endpoints** section. It should look like `https://ingress.eu-west-1.aws.dash0.com/v1/traces`. Make sure it ends with `/v1/traces/` 3. Add the endpoint and the auth token to the Checkly integration settings. The auth token is added as an HTTP header named `Authorization` with the value of `Bearer ` and your auth token. Fill in the values into Checkly

4. Observe Spans appearing in Dash0 on the `Tracing` tab. Traces displayed in Dash0

Find all the details in the [Dash0 documentation](https://www.dash0.com/documentation). # Exporting Traces to Grafana Tempo Source: https://www.checklyhq.com/docs/resolve/traces/export/grafana-tempo Learn how to export traces from Checkly to Grafana Tempo. You can connect the Traces export to Grafana using the OpenTelemetry Integration and Grafana Tempo. 1. Make sure that you have Tempo running on your Grafana Cloud Instance at `yourOrganization.grafana.net`. Find it at **Connections > Search for Tempo > Check if Tempo Data source is marked as installed**. 2. In Grafana Cloud (`grafana.com/orgs/yourOrganization`), head over to **Connections** > **Add new connection** > **OpenTelemetry (OTLP)**. Add an OTLP connection in Grafana Cloud

3. Copy the endpoint URL and append `v1/traces` to it. Grafana OTLP endpoint

It should be similar to:`https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/traces`. 4. Generate an OTLP Token, and copy both the Instance ID and the OTLP Token as well.

5. Head back to the [Traces Settings page](https://app.checklyhq.com/settings/account/traces) on Checkly. * Enable exporting traces and add the endpoint URL from step 3. * and in the **Headers** section, specify the HTTP Header: `Authorization` and `Basic instance:token`, where `instance:token` is base64 encoded. You can use an online tool like [base64encode.net](https://www.base64encode.net/) to encode your instance and token. Export Traces configuration

2. Back in your Grafana Cloud Instance (for example danube.grafana.net), head over to "Explore", select the *Tempo* source that is named `grafanacloud-yourOrganization-traces`: Select Tempo traces data source

Now, click **Search** to see the table of Traces received. The ones exported by Checkly, have `checkly` in the service column. See Checkly Traces in Grafana

Find more the details in the [Grafana OpenTelemetry documentation](https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/?pg=traces\&plcmt=hero-btn-2#before-you-begin). # Exporting Traces to Honeycomb Source: https://www.checklyhq.com/docs/resolve/traces/export/honeycomb Learn how to export traces from Checkly to Honeycomb. 1. Grab the relevant API endpoint from [the Honeycomb documentation](https://docs.honeycomb.io/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint). It should look like `https://api.honeycomb.io/v1/traces`. 2. Grab your Honeycomb Ingest API key from the **Account** > **Team settings** > **Environments and API keys** section. 3. Add the endpoint the Checkly integration settings and provide the API key as an HTTP header `x-honeycomb-team` with the value of your API key. # Exporting Traces to New Relic Source: https://www.checklyhq.com/docs/resolve/traces/export/new-relic Learn how to export traces from Checkly to New Relic. 1. Make sure you have the **Traces** feature enabled in your New Relic account. 2. Create an *Ingest - License* type API key in the **Administration** > **API keys** section. 3. Grab the right endpoint URL for your New Relic account. It should look like `https://otlp.nr-data.net/v1/traces`. Check [the full list of all New Relic OTel endpoints](https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/get-started/opentelemetry-set-up-your-app/#ports-and-endpoints). 4. Add the endpoint and the API key to the Checkly integration settings. The API key is added as an HTTP header named `api-key` with the value of your API key. Find all the details in the [New Relic OpenTelemetry documentation](https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/get-started/opentelemetry-set-up-your-app/). # How It Works Source: https://www.checklyhq.com/docs/resolve/traces/how-it-works Understand how Checkly traces work with OpenTelemetry to provide comprehensive observability. Checkly Traces use OpenTelemetry to correlate synthetic check results with your application's backend traces. The diagram below shows how the dataflows connect to Checkly's monitoring and alerting pipeline. Checkly OpenTelemetry diagram

## The Flow When enabling the integration, Checkly automatically instruments all HTTP requests made by your checks with `traceparent` and `tracestate` headers. These HTTP requests hit your web application and/or API when running checks. All synthetic check types, including API, Browser and Multistep checks, include these headers: * **traceparent**: W3C compliant trace ID generated for each HTTP request that is part of a check. For Browser checks and Multistep checks, there can be multiple requests instrumented. * **tracestate**: Set to `checkly=true` to identify Checkly as the vendor that generated the trace Together, these headers propagate the trace context along the request chain. This is the most basic way of tying a synthetic check to your backend traces, though it won't provide much context about the check run itself, like the check name or location. You can configure your 3rd party backend in the Checkly UI so we can export every check result as a span to your backend. This correlates all backend spans with the check run that triggered them, together with full context of the check run. Check details are stored in the `checkly` namespace as attributes on the span: ```yaml theme={null} checkly.check.name: "ACME homepage" checkly.check.id: "438481ea-0eab-43d6-8932-ab51bd0d49d6" checkly.check.type: "browser" checkly.check.location: "eu-west-1" checkly.check.tags: ["production", "checkout"] ``` This provides rich context about the run location, check name, check type, and more. `checkly.check.tags` carries the check's tags as a string array — the check's own tags first, then tags inherited from its group, deduplicated. Use it to filter or group traces by the same labels you organize your checks by, such as a team or workflow tag. Tag caveats: * The attribute is only present when the check has tags. At most 20 tags are exported per span, and a tag longer than 128 characters is omitted entirely rather than truncated. * Attributes are set on Checkly's root span only — they do not propagate to your application's own spans within the trace. To find a whole trace by tag, filter on the root span, then use its trace ID. **No 3rd party backend needed.** This is what kicks Checkly's OpenTelemetry integration into high gear, and it works even if you don't have a 3rd party OTel backend already set up. By adding the correct OTel libraries and simple filter statements, you can send ONLY the traces related to your Checkly checks back to Checkly: * **Endpoint**: `otel.eu-west-1.checklyhq.com/v1/traces` * **Filter**: Only traces marked `tracestate: checkly=true` * **Result**: Traces appear in Checkly UI alongside check results Checkly's trace ingestion endpoint is a standard OTel backend endpoint that only accepts traces related to synthetic checks. This keeps your OpenTelemetry setup lean while providing full trace correlation benefits. When your check breaks, you get an alert that points to a check result containing a full trace of the check run, including all backend spans: * Complete trace of the check run * All backend spans included * Quick root cause identification * No need to jump between tools or dashboards This allows you to quickly identify issues without switching between different monitoring tools. Steps 3 and 4 provide the most value - complete trace visibility without requiring a 3rd party OpenTelemetry backend. # HTTPS vs GRPC Traces Source: https://www.checklyhq.com/docs/resolve/traces/import/https-grpc Learn how to import traces from HTTPS/GRPC Checkly's OTel ingestion endpoint supports both HTTP and gRPC protocols. The gRPC protocol is the "native" protocol for OTel, but in 2020 an alternative HTTP protocol was introduced due to some of the downsides gRPC has: * gRPC tends to come with bigger dependencies in your code base. * gRPC uses HTTP/2 as its transport layer, which might not be 100% supported in your network infra. Think firewalls, proxies and load balancers. However, for larger payload and higher throughput scenarios, gRPC is the way to go. It's more efficient and has a smaller connection overhead. Different SDKs and languages have different levels of support for gRPC and HTTP. For example, the HTTP protocol is the default for Node.js, but gRPC is the default for Go. In all cases, we recommend using the default protocol for your language and SDK. # Missing Spans Source: https://www.checklyhq.com/docs/resolve/traces/import/missing-spans Learn how to import traces from OpenTelemetry This guide will walk you through the process of identifying missing spans, understanding their root causes, and implementing solutions to ensure your OpenTelemetry setup with Checkly provides complete and accurate tracing information. ## Missing spans Missing spans, also known as orphan spans, are segments of a trace that are disconnected from either their parent spans or from the root span. An instrumented distributed system will emit spans, creating a hierarchical chain that follows the flow of the request through the different components and services. When a span is missing, they create a hole in this chain, making it difficult to diagnose issues and understand the full picture. If this happens, you will notice a span missing from the visualization in Checkly. Checkly uses contextual span data—such as the trace it belongs to and its parent span—to display any missing pieces in the span chain. This helps you identify which application or service may be incorrectly instrumented. Trace with missing spans

## Troubleshooting missing spans Missing spans can happen due to various reasons. Whether you're sending data to Checkly directly from your applications and services, or through the OpenTelemetry Collector, these are steps to confirm everything is properly set up. ### Verify sampling strategy Using Checkly Traces, we recommend sampling for the tracestate header `checkly=true`, which will reduce your Egress/Ingress costs and ensure that all the spans that were originated through a check are there. #### Sampling in the OpenTelemetry Collector * **Step 1**. Make sure that you're filtering out non-checkly spans: ```yaml theme={null} processors: batch: filter/checkly: error_mode: ignore traces: span: # filter out spans in which tracestate is not "checkly=true" - 'trace_state["checkly"] != "true"' ``` * **Step 2**. Confirm that your Collector pipeline uses this processor: ```yaml theme={null} service: pipelines: traces: processors: [filter/checkly, batch] ``` #### Sampling directly in your applications' code. Choose the specific [instrumentation language guide](/docs/resolve/traces/instrumentation/nodejs) and review your configuration against Step 2 in the guides. ### Review OpenTelemetry Exporter configuration Ensure the right Authorization keys and endpoints are in use. #### Exporting through the OpenTelemetry Collector * **Step 1**. Make sure that your exporter uses the right endpoint authorization: ```yaml theme={null} exporters: otlp/checkly: endpoint: "otel.eu-west-1.checklyhq.com:4317" headers: authorization: "${env:CHECKLY_OTEL_API_KEY}" ``` * **Step 2**. Confirm that your Collector pipeline uses this exporter: ```yaml theme={null} service: pipelines: traces: exporters: [otlp/checkly] ``` #### Exporting directly from your Application's code Confirm your environment variables use the correct endpoint and authorization: ```bash Terminal theme={null} OTEL_EXPORTER_OTLP_HEADERS="authorization=" OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" ``` ### Checkly Private Location setup When setting up Traces with a [Checkly Private Location](/docs/platform/private-locations/overview): * **Step 1**. Ensure your [Checkly Agent](https://hub.docker.com/r/checkly/agent) version is at least 3.3.5. * **Step 2**. Review your internal Firewall rules: * Identify the ports used by OpenTelemetry, commonly: * Port 4317 for gRPC protocol * Port 4318 for HTTP protocol * Access your firewall settings and add rules to allow outgoing traffic on the identified port. For example, to allow HTTP traces to be sent out: ```bash Terminal theme={null} A OUTPUT -p tcp --dport 4318 -j ACCEPT ``` ### Incomplete instrumentation To see the full picture, and identify the root cause of a problem faster, ensure your applications and services are [instrumented with the OpenTelemetry SDK](/docs/resolve/traces/instrumentation/nodejs). # Setting Up The OpenTelemetry Collector Source: https://www.checklyhq.com/docs/resolve/traces/import/open-telemetry Learn how to import traces from OpenTelemetry Using the OpenTelemetry collector is the best way to manage and route telemetry data to Checkly. It requires a little setting up, but it's worth it. ## Prerequisites * An OpenTelemetry collector running and configured. If you need to set up OpenTelemetry for the first time, please use [the Open Telemetry Collector getting started documentation](https://opentelemetry.io/docs/collector/installation/). ## Step 1: Update your config Sending traces to Checkly is very simple with the [OpenTelemetry collector](/docs/learn/opentelemetry/what-is-the-otel-collector/). There are three sections you need to add to your collector configuration file: 1. Create a `filter` that will remove all spans that don't have `checkly=true` in the trace state. 2. Create a new `exporter` with the Checkly API endpoint and API key as an environment variables. 3. Hook the `exporter` and `filter` into a new trace pipeline. ```yaml YAML theme={null} receivers: otlp: protocols: http: grpc: processors: batch: filter/checkly: error_mode: ignore traces: span: # remove all spans that the trace state doesn't have an object # which key is "checkly" and value is "true" - 'trace_state["checkly"] != "true"' exporters: otlp/checkly: endpoint: "otel.eu-west-1.checklyhq.com:4317" headers: authorization: "${env:CHECKLY_OTEL_API_KEY}" service: pipelines: traces: receivers: [otlp] processors: [filter/checkly, batch] exporters: [otlp/checkly] ``` ## Step 2: Restart your collector Grab your **OTel API key** in the *Send traces* section of the [Open Telemetry Integration page in the Checkly app](https://app.checklyhq.com/settings/account/open-telemetry). Now, export your API key in your shell by setting the `CHECKLY_OTEL_API_KEY` environment variable. ```bash Terminal theme={null} export CHECKLY_OTEL_API_KEY="" ``` Now, restart your collector with the updated configuration file. If you are using a Dockerized version of the OTel collector, don't forget to pass in the environment variables, e.g. ```bash Terminal theme={null} docker run \ -e CHECKLY_OTEL_API_KEY \ ... ``` ## Step 3: Verify your setup You are done. Any traces ingested by your collector that are triggered by a Checkly synthetic check will now be sent to Checkly via the new pipeline. If you are not using the OpenTelemetry collector, you can also [send traces directly to Checkly by instrumenting your application code](/docs/resolve/traces/instrumentation/nodejs). # Importing Traces To Checkly Source: https://www.checklyhq.com/docs/resolve/traces/import/overview Send OpenTelemetry traces from your infrastructure to Checkly so you can drill into why a check failed or degraded, right from the check result. Send your infrastructure OpenTelemetry traces to Checkly. Inspect and drill down to why a check failed or degraded. OTEL Collector Diagram with Checkly Traces

OTEL Collector Diagram with Checkly Traces

In [the Traces tab, in your Checkly account settings](https://app.checklyhq.com/settings/account/traces): 1. Flip the toggle to import traces in the [Traces settings page](https://app.checklyhq.com/settings/account/traces). 2. Get the trace endpoint URL and the API key. Provide them to your OpenTelemetry collector so that it sends traces back to Checkly. Provide URL and header with authentication

Provide URL and header with authentication

3. Set up your OpenTelemetry collector to only send relevant traces to Checkly. [Setting up the OpenTelemetry Collector](/docs/resolve/traces/import/open-telemetry) After you've set this up, you should start seeing traces on the [Traces page](https://app.checklyhq.com/traces). Moreover, when opening a check, you'll see a Traces section within it. Check result with OpenTelemetry trace

# Django Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/django Learn how to instrument your Django code with OpenTelemetry. This guide will help you instrument your Django application(s) with OpenTelemetry and send traces to Checkly. ## Step 1: Install the OpenTelemetry SDK Install the relevant OpenTelemetry packages: ```bash Terminal theme={null} pip install opentelemetry-sdk \ opentelemetry-exporter-otlp \ opentelemetry-instrumentation-django \ requests ``` ## Step 2: Initialize the instrumentation Based on the web server you are using, you need to initialize the OpenTelemetry SDK and set up the necessary instrumentation. ### Gunicorn Open your `gunicorn.config.py` file and add the following code. This will initialize the OpenTelemetry SDK and set up the necessary instrumentation. Notice the `HttpHeaderSampler` class. This is a custom, head-based sampler that will only sample spans that are generated by Checkly by inspecting the trace state. This way you only pay for the egress traffic generated by Checkly and not for any other traffic. Note the code in the `post_fork` function. This will instrument your Django app with OpenTelemetry. ```python gunicorn.config.py theme={null} from opentelemetry import trace from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.sampling import Sampler, SamplingResult, Decision from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.instrumentation.django import DjangoInstrumentor from requests import request class HttpHeaderSampler(Sampler): def get_description(self) -> str: return "HttpHeaderSampler" def should_sample( parent_context, trace_id, name, kind=None, attributes=None, links=None, trace_state=None ) -> SamplingResult: if request.headers.get('tracestate') == 'checkly=true': return SamplingResult(Decision.RECORD_AND_SAMPLE) else: return SamplingResult(Decision.DROP ) trace.set_tracer_provider(TracerProvider(sampler=HttpHeaderSampler())) trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) def post_fork(server, worker): DjangoInstrumentor().instrument(tracer_provider=trace.get_tracer_provider()) ``` ### uWSGI When using uWSGI, you can use the `post_fork` hook to instrument your Django app with OpenTelemetry. Note you will need to install the `uwsgidecorators` package. ```bash Terminal theme={null} pip install uwsgidecorators ``` And then add the following code to your `wsgi.py` file. ```python wsgi.py theme={null} from uwsgidecorators import postfork from opentelemetry import trace from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.sampling import Sampler, SamplingResult, Decision from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.instrumentation.django import DjangoInstrumentor from requests import request class HttpHeaderSampler(Sampler): def get_description(self) -> str: return "HttpHeaderSampler" def should_sample( parent_context, trace_id, name, kind=None, attributes=None, links=None, trace_state=None ) -> SamplingResult: if request.headers.get('tracestate') == 'checkly=true': return SamplingResult(Decision.RECORD_AND_SAMPLE) else: return SamplingResult(Decision.DROP ) trace.set_tracer_provider(TracerProvider(sampler=HttpHeaderSampler())) trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) @postfork def init_tracing(): DjangoInstrumentor().instrument(tracer_provider=trace.get_tracer_provider()) ``` ## Step 3: Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

Now, export your API key in your shell by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable. ```bash theme={null} export OTEL_EXPORTER_OTLP_HEADERS="authorization=" ``` Next, export the endpoint for the region you want to use and give your service a name. ```bash theme={null} export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" export OTEL_SERVICE_NAME="your-service-name" ``` Then, explicitly set the protocol to use for the OTLP exporter. ```bash theme={null} export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` We are using the standard OpenTelemetry environment variables here to configure the OTLP exporter. | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_HEADERS` | The `Authorization` HTTP header containing your Checkly OTel API key. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The Checkly OTel API endpoint for the region you want to use. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The protocol to use for the OTLP exporter. | | `OTEL_SERVICE_NAME` | The name of your service to identify it among the spans in the web UI. | Finally, start your app using the relevant command: For Gunicorn, your startup command will look similar to ```bash Terminal theme={null} gunicorn myapp.wsgi -c gunicorn.config.py --workers 2 --threads 2 ``` For uWSGI, your startup command will look similar to ```bash Terminal theme={null} uwsgi --http :8000 --wsgi-file myapp.wsgi --master --processes 4 --threads 2 ``` 🎉 You are done. Any interactions with your app that are triggered by a Checkly synthetic monitoring check will now generate traces, which are sent back to Checkly and displayed in the Checkly UI. # .NET Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/dotnet Learn how to instrument your .NET code with OpenTelemetry. This guide shows you how to instrument an ASP.NET Core application and send traces from Checkly-triggered requests to Checkly. For other .NET application types and instrumentation libraries, refer to the [OpenTelemetry .NET documentation](https://opentelemetry.io/docs/languages/dotnet/). * A .NET application that receives requests from a Checkly check. * An OTel API key and endpoint from the [OpenTelemetry Integration page](https://app.checklyhq.com/settings/account/open-telemetry) in the Checkly app. ## Install OpenTelemetry packages Install the OpenTelemetry SDK, OTLP exporter, and ASP.NET Core instrumentation: ```bash Terminal theme={null} dotnet add package OpenTelemetry.Extensions.Hosting dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol dotnet add package OpenTelemetry.Instrumentation.AspNetCore ``` To trace outgoing `HttpClient` requests as part of the same trace, also install the HTTP client instrumentation: ```bash Terminal theme={null} dotnet add package OpenTelemetry.Instrumentation.Http ``` ## Add the Checkly sampler Create `ChecklySampler.cs` in your project: ```csharp ChecklySampler.cs theme={null} using System; using OpenTelemetry.Trace; public sealed class ChecklySampler : Sampler { public override SamplingResult ShouldSample(in SamplingParameters samplingParameters) { return HasChecklyTraceState(samplingParameters.ParentContext.TraceState) ? new SamplingResult(SamplingDecision.RecordAndSample) : new SamplingResult(SamplingDecision.Drop); } private static bool HasChecklyTraceState(string? traceState) { if (string.IsNullOrWhiteSpace(traceState)) { return false; } foreach (var entry in traceState.Split(',')) { var keyValue = entry.Trim().Split('=', 2); if (keyValue.Length == 2 && string.Equals(keyValue[0], "checkly", StringComparison.Ordinal) && string.Equals(keyValue[1], "true", StringComparison.Ordinal)) { return true; } } return false; } } ``` Checkly adds `checkly=true` to the W3C `tracestate` header on requests from its checks. This sampler records only traces that carry that value, so your application exports Checkly-triggered traces rather than all application traffic. ## Configure tracing Register the sampler, the ASP.NET Core instrumentation, and the OTLP exporter in `Program.cs`: ```csharp Program.cs theme={null} using OpenTelemetry.Trace; var builder = WebApplication.CreateBuilder(args); builder.Services.AddOpenTelemetry() .WithTracing(tracing => tracing .SetSampler(new ChecklySampler()) .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddOtlpExporter()); var app = builder.Build(); app.Run(); ``` Remove `.AddHttpClientInstrumentation()` if you did not install `OpenTelemetry.Instrumentation.Http`. ## Set Checkly connection details Configure the OTLP exporter with standard OpenTelemetry environment variables. Use the endpoint for your Checkly region from the OpenTelemetry Integration page. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_HEADERS="authorization=" export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" export OTEL_SERVICE_NAME="your-service-name" ``` The .NET OTLP exporter uses gRPC by default. Set `OTEL_EXPORTER_OTLP_PROTOCOL` to `http/protobuf` to send traces to the Checkly HTTP endpoint. Keep the API key outside source control. Configure it through your deployment environment or secret manager. ## Verify the integration Start your application, then run a Checkly check that calls it. The check-triggered request and its downstream spans should appear with the check result in Checkly. For more information about how Checkly correlates check results and traces, see [How Traces work](/docs/resolve/traces/how-it-works/). # Express Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/express Learn how to instrument your Express code with OpenTelemetry. This guide will help you instrument your Express application(s) with OpenTelemetry and send traces to Checkly. The steps are largely the same as instrumenting any Node.js application with OpenTelemetry. ## Step 1: Install the OpenTelemetry packages Install the relevant OpenTelemetry packages: ```bash Terminal theme={null} npm install --save \ @opentelemetry/sdk-node \ @opentelemetry/exporter-trace-otlp-proto \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-trace-base \ @opentelemetry/api ``` ## Step 2: Initialize the instrumentation Create a file called `tracing.js` at the root of your project and add the following code: ```javascript tracing.js theme={null} const { NodeSDK } = require('@opentelemetry/sdk-node') const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto') const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node') const { BatchSpanProcessor, SamplingDecision } = require('@opentelemetry/sdk-trace-base') const { trace } = require('@opentelemetry/api') const exporter = new OTLPTraceExporter({ timeoutMillis: 2000, }) const sdk = new NodeSDK({ instrumentations: [getNodeAutoInstrumentations()], spanProcessors: new BatchSpanProcessor(exporter), sampler: { shouldSample: (context, traceId, spanName, spanKind, attributes, links) => { const isChecklySpan = trace.getSpan(context)?.spanContext()?.traceState?.get('checkly') if (isChecklySpan) { return { decision: SamplingDecision.RECORD_AND_SAMPLED } } else { return { decision: SamplingDecision.NOT_RECORD } } }, }, }) sdk.start() process.on('SIGTERM', () => { sdk.shutdown() .then(() => console.log('OTel Tracing terminated')) .catch((error) => console.log('Error terminating OTel tracing', error)) .finally(() => process.exit(0)) }) ``` Notice the `sampler` configuration. This is a custom, **head-based sampler** that will only sample spans that are generated by Checkly by inspecting the trace state. This way you only pay for the egress traffic generated by Checkly and not for any other traffic. ## Step 3: Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

Now, export your API key in your shell by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_HEADERS="authorization=" ``` Next, export the endpoint for the region you want to use and give your service a name. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" export OTEL_SERVICE_NAME="your-service-name" ``` Then, explicitly set the protocol to use for the OTLP exporter. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` We are using the standard OpenTelemetry environment variables here to configure the OTLP exporter. | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_HEADERS` | The `Authorization` HTTP header containing your Checkly OTel API key. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The Checkly OTel API endpoint for the region you want to use. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The protocol to use for the OTLP exporter. | | `OTEL_SERVICE_NAME` | The name of your service to identify it among the spans in the web UI. | Finally, start your app with the Flask CLI: ```bash Terminal theme={null} flask run ``` 🎉 You are done. Any interactions with your app that are triggered by a Checkly synthetic monitoring check will now generate traces, which are sent back to Checkly and displayed in the Checkly UI. # Go / Golang Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/golang Learn how to instrument your Go/Golang code with OpenTelemetry. This guide will help you instrument the `net/http` stack of your Go application(s) with OpenTelemetry and send traces to Checkly. ## Step 1: Install the OpenTelemetry SDK Install the relevant OpenTelemetry packages: ```bash Terminal theme={null} go get "go.opentelemetry.io/otel" \ "go.opentelemetry.io/otel/propagation" \ "go.opentelemetry.io/otel/sdk/trace" \ "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \ "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" ``` ## Step 2: Set up SDK Create a file called `tracing.go` at the root of your project and add the following code: ```go tracing.go theme={null} package main import ( "context" "errors" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/trace" ) func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) { var shutdownFuncs []func(context.Context) error shutdown = func(ctx context.Context) error { var err error for _, fn := range shutdownFuncs { err = errors.Join(err, fn(ctx)) } shutdownFuncs = nil return err } handleErr := func(inErr error) { err = errors.Join(inErr, shutdown(ctx)) } prop := newPropagator() otel.SetTextMapPropagator(prop) tracerProvider, err := newTraceProvider() if err != nil { handleErr(err) return } shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown) otel.SetTracerProvider(tracerProvider) return } func newPropagator() propagation.TextMapPropagator { return propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, propagation.Baggage{}, ) } func newTraceProvider() (*trace.TracerProvider, error) { traceClient := otlptracehttp.NewClient() var traceExporter, err = otlptrace.New(context.Background(), traceClient) if err != nil { return nil, err } bsp := trace.NewBatchSpanProcessor(traceExporter) tracerProvider := trace.NewTracerProvider( trace.WithSampler(trace.AlwaysSample()), trace.WithSpanProcessor(bsp), ) return tracerProvider, nil } ``` ## Step 3: Initialize the instrumentation Add or adapt the following code to your `main.go` file. The key parts are as follows: 1. Wrap the default `http.Handler` with the`otelhttp.NewHandler` function. 2. Add the `otelChecklyFilter` function to filter out the requests that should be traced and send to Checkly. ```go main.go theme={null} package main import ( "context" "errors" "log" "net" "net/http" "os" "os/signal" "strings" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" ) func main() { if err := run(); err != nil { log.Fatalln(err) } } func run() (err error) { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) defer stop() otelShutdown, err := setupOTelSDK(ctx) if err != nil { return } defer func() { err = otelShutdown(context.Background()) }() handler := http.Handler(http.DefaultServeMux) wrappedHandler := otelhttp.NewHandler(handler, "", otelhttp.WithFilter(otelChecklyFilter)) srv := &http.Server{ Addr: ":8080", BaseContext: func(_ net.Listener) context.Context { return ctx }, Handler: wrappedHandler, } srvErr := make(chan error, 1) go func() { srvErr <- srv.ListenAndServe() }() select { case err = <-srvErr: return case <-ctx.Done(): stop() } err = srv.Shutdown(context.Background()) return } func otelChecklyFilter(req *http.Request) bool { header := req.Header.Get("tracestate") return header != "" && strings.Contains(header, "checkly=true") } ``` ## Step 4: Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

Now, export your API key in your shell by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_HEADERS="authorization=" ``` Next, export the endpoint for the region you want to use and give your service a name. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" export OTEL_SERVICE_NAME="your-service-name" ``` Then, explicitly set the protocol to use for the OTLP exporter. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` We are using the standard OpenTelemetry environment variables here to configure the OTLP exporter. | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_HEADERS` | The `Authorization` HTTP header containing your Checkly OTel API key. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The Checkly OTel API endpoint for the region you want to use. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The protocol to use for the OTLP exporter. | | `OTEL_SERVICE_NAME` | The name of your service to identify it among the spans in the web UI. | Then run you app as usual: ```bash Terminal theme={null} go run . ``` # Java Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/java Learn how to instrument your Java code with OpenTelemetry. Java docs coming soon. For now, check the canonical OpenTelemetry Java documentation at [opentelemetry.io](https://opentelemetry.io/docs/languages/java/) or the great content from [signoz.io](https://signoz.io/docs/instrumentation/java/). # Laravel Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/laravel Learn how to instrument your Laravel code with OpenTelemetry. This guide will help you instrument your Laravel application(s) with OpenTelemetry and send traces to Checkly. ## Step 1: Install the OpenTelemetry extension ``` pecl install opentelemetry ``` Check if the `opentelemetry` extension is properly installed and enabled: ```bash theme={null} php --ri opentelemetry ``` You should see something similar to: ```bash theme={null} opentelemetry opentelemetry hooks => enabled extension version => 1.0.3 Directive => Local Value => Master Value opentelemetry.conflicts => no value => no value opentelemetry.validate_hook_functions => On => On opentelemetry.allow_stack_extension => Off => Off ``` If you are not getting a similar output, check your `php.ini` file and make sure a line is added similar to this: ``` extension=/some/path/opentelemetry.so ``` Now install the automatic Laravel instrumentation. ```bash Terminal theme={null} composer require open-telemetry/opentelemetry-auto-laravel ``` ## Step 2: Install the OpenTelemetry SDK ```bash Terminal theme={null} composer require \ open-telemetry/opentelemetry \ open-telemetry/sdk \ open-telemetry/api \ open-telemetry/opentelemetry-auto-slim \ open-telemetry/opentelemetry-auto-laravel \ open-telemetry/exporter-otlp ``` Now install the automatic Laravel instrumentation. ```bash theme={null} composer require open-telemetry/opentelemetry-auto-laravel ``` ## Step 3: Initialize the instrumentation This example shows all the code you will need to add to your `index.php`. Notice the `ChecklySampler` class. This is a custom, head-based sampler that will only sample spans that are generated by Checkly by inspecting the trace state. This way you only pay for the egress traffic generated by Checkly and not for any other traffic. ```php index.php theme={null} getContext(); $traceState = $parentSpanContext->getTraceState(); if ($traceState !== '' && $traceState !== null && str_contains($traceState, 'checkly')) { return new SamplingResult( SamplingResult::RECORD_AND_SAMPLE, [], $traceState ); } else { return new SamplingResult( SamplingResult::DROP, [], $traceState ); } } public function getDescription(): string { return 'ChecklySampler'; } } $exporter = (new SpanExporterFactory())->create(); $tracerProvider = new TracerProvider( new BatchSpanProcessor($exporter, ClockFactory::getDefault()), new ChecklySampler() ); Sdk::builder() ->setTracerProvider($tracerProvider) ->setPropagator(TraceContextPropagator::getInstance()) ->setAutoShutdown(true) ->buildAndRegisterGlobal(); // Bootstrap Laravel and handle the request... (require_once __DIR__.'/../bootstrap/app.php') ->handleRequest(Request::capture()); ``` ## Step 3: Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

Now, export your API key in your shell by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_HEADERS="authorization=" ``` Next, export the endpoint for the region you want to use and give your service a name. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" export OTEL_SERVICE_NAME="your-service-name" ``` Then, explicitly set the protocol to use for the OTLP exporter. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` We are using the standard OpenTelemetry environment variables here to configure the OTLP exporter. | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_HEADERS` | The `Authorization` HTTP header containing your Checkly OTel API key. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The Checkly OTel API endpoint for the region you want to use. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The protocol to use for the OTLP exporter. | | `OTEL_SERVICE_NAME` | The name of your service to identify it among the spans in the web UI. | Finally, start your app: ```bash Terminal theme={null} php artisan serve ``` 🎉 You are done. Any interactions with your app that are triggered by a Checkly synthetic monitoring check will now generate traces, which are sent back to Checkly and displayed in the Checkly UI. # Next.js Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/nextjs Learn how to instrument your Next.js code with OpenTelemetry. Next.js is now shipping with an experimental OpenTelemetry integration. This guide will help you instrument your Next.js application(s) with OpenTelemetry and send traces to Checkly. ## Before you start You need: * A Next.js application using at least [Next.js 14](https://nextjs.org/blog/next-14). ## Step 1. Install the `@vercel/otel` package For Next.js, Vercel has created a wrapper that should get you going very quickly. We're just adding some extra packages so we can filter out the Checkly Traces. ```bash Terminal theme={null} npm install --save \ @vercel/otel \ @opentelemetry/api \ @opentelemetry/sdk-trace-base \ @opentelemetry/exporter-trace-otlp-http ``` Make sure to install v1.9.1 or later of the `@vercel/otel` package as it contains a critical bug fix for Vercel's Edge runtime. ## Step 2. \[Only in Next.js 14]: Initialize the instrumentation Set the `instrumentationHook` flag to `true` in your Next.js configuration file. This will enable the OpenTelemetry instrumentation. ```js next.config.js theme={null} /** @type {import('next').NextConfig} */ const nextConfig = { experimental: { instrumentationHook: true } } module.exports = nextConfig ``` ## Step 3. Set up the instrumentation Create an `instrumentation.ts` file in the root of your project, or, place it in the `src` directory if you are using one. And add the following code: ```ts instrumentation.ts theme={null} import { registerOTel } from '@vercel/otel' import { SamplingDecision } from '@opentelemetry/sdk-trace-base' import { trace, Context } from '@opentelemetry/api' export function register() { registerOTel({ serviceName: 'acme-next-app', traceExporter: 'auto', spanProcessors: ['auto'], traceSampler: { shouldSample: (context: Context) => { const isChecklySpan = trace.getSpan(context)?.spanContext()?.traceState?.get('checkly') if (isChecklySpan) { console.log('Sampling decision for Checkly span:', SamplingDecision.RECORD_AND_SAMPLED) return { decision: SamplingDecision.RECORD_AND_SAMPLED } } else { console.log('Sampling decision for non-Checkly span:', SamplingDecision.NOT_RECORD) return { decision: SamplingDecision.NOT_RECORD } } }, }, }) } ``` Notice that we are using the default exporter and span processor from the `@vercel/otel` package, which is configured by using the `auto` value. This package is compatible with Vercel's Edge runtime, in contrast to the `@opentelemetry/exporter-trace-otlp-http` package, which is not. If you are not using Vercel or don't use Vercel's Edge runtime, you can also use the `@opentelemetry/exporter-trace-otlp-http` and just follow [the regular Node.js instrumentation guide](/docs/resolve/traces/instrumentation/nodejs#step-2-initialize-the-instrumentation). This is however not required, the above configuration should work just fine on Node.js and Edge runtimes. Also notice the `sampler` configuration. This is a custom, **head-based sampler** that will only sample spans that are generated by Checkly by inspecting the trace state. This way you only pay for the egress traffic generated by Checkly and not for any other traffic. ## Step 5 (Local development). Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

### 5.2. Run a check and confirm it works! Schedule a check, you should see the Vercel logs reporting `Sampling decision for Checkly span: #` and soon the spans will show up in your check results page as well. ## Further reading Have a look at the official Next.js docs on [how to enable the experimental OpenTelemetry integration](https://nextjs.org/docs/app/building-your-application/optimizing/open-telemetry). # Node.js Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/nodejs Learn how to instrument your Node.js code with OpenTelemetry. This guide will help you instrument your Node.js application(s) with OpenTelemetry and send traces to Checkly. ## Step 1: Install the OpenTelemetry packages Install the relevant OpenTelemetry packages: ```bash Terminal theme={null} npm install --save \ @opentelemetry/sdk-node \ @opentelemetry/exporter-trace-otlp-proto \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-trace-base \ @opentelemetry/api ``` ## Step 2: Initialize the instrumentation Create a file called `tracing.js` at the root of your project and add the following code: ```javascript tracing.js theme={null} const { NodeSDK } = require('@opentelemetry/sdk-node') const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto') const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node') const { BatchSpanProcessor, SamplingDecision } = require('@opentelemetry/sdk-trace-base') const { trace } = require('@opentelemetry/api') const exporter = new OTLPTraceExporter({ timeoutMillis: 2000, }) const sdk = new NodeSDK({ instrumentations: [getNodeAutoInstrumentations()], spanProcessors: [new BatchSpanProcessor(exporter)], sampler: { shouldSample: (context, traceId, spanName, spanKind, attributes, links) => { const isChecklySpan = trace.getSpan(context)?.spanContext()?.traceState?.get('checkly') if (isChecklySpan) { return { decision: SamplingDecision.RECORD_AND_SAMPLED } } else { return { decision: SamplingDecision.NOT_RECORD } } }, }, }) sdk.start() process.on('SIGTERM', () => { sdk.shutdown() .then(() => console.log('OTel Tracing terminated')) .catch((error) => console.log('Error terminating OTel tracing', error)) .finally(() => process.exit(0)) }) ``` Notice the `sampler` configuration. This is a custom, **head-based sampler** that will only sample spans that are generated by Checkly by inspecting the trace state. This way you only pay for the egress traffic generated by Checkly and not for any other traffic. ## Step 3: Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

Now, export your API key in your shell by setting the `OTEL_EXPORTER_OTLP_HEADERS` environment variable. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_HEADERS="authorization=" ``` Next, export the endpoint for the region you want to use and give your service a name. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com" export OTEL_SERVICE_NAME="your-service-name" ``` Then, explicitly set the protocol to use for the OTLP exporter. ```bash Terminal theme={null} export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` We are using the standard OpenTelemetry environment variables here to configure the OTLP exporter. | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_HEADERS` | The `Authorization` HTTP header containing your Checkly OTel API key. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | The Checkly OTel API endpoint for the region you want to use. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The protocol to use for the OTLP exporter. | | `OTEL_SERVICE_NAME` | The name of your service to identify it among the spans in the web UI. | Then start your app with the extra `-r` flag to load the `tracing.js` file before any other files are loaded. ```bash Terminal theme={null} node -r ./tracing.js index.js ``` 🎉 You are done. Any interactions with your app that are triggered by a Checkly synthetic monitoring check will now generate traces, which are sent back to Checkly and displayed in the Checkly UI. ## Reducing noise in the auto Node.js instrumentation We found the Node.js auto-instrumentation a bit noisy. There can be a lot of file i/o and a ton of DNS calls you might not be interested in. Luckily, you can easily tweak that by providing some extra options to the `getNodeAutoInstrumentations()` function. We use the following configuration to filter out some of the noise: ```javascript tracing.js theme={null} instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false, }, '@opentelemetry/instrumentation-net': { enabled: false, }, '@opentelemetry/instrumentation-dns': { enabled: false, }, '@opentelemetry/instrumentation-http': { enabled: true, }, })] ``` # Ruby on Rails Source: https://www.checklyhq.com/docs/resolve/traces/instrumentation/ruby-on-rails Learn how to instrument your Ruby on Rails code with OpenTelemetry. This guide will help you instrument your Ruby on Rails application(s) with OpenTelemetry and send traces to Checkly. Although this guide is for [Ruby on Rails](https://rubyonrails.org/), the steps are largely the same as instrumenting any Ruby application with OpenTelemetry. This guide assumes you have a basic understanding of Ruby on Rails and already have a working Rails application. ## Step 1: Install the OpenTelemetry packages Go to the root of your Rails app and add the basic OpenTelemetry SDK, OTLP exporter and instrumentation gems to your Gemfile: ```bash Terminal theme={null} bundle add opentelemetry-sdk \ opentelemetry-exporter-otlp \ opentelemetry-instrumentation-all ``` This should add the following lines: ```ruby Gemfile theme={null} gem "opentelemetry-sdk", "~> 1.4" gem "opentelemetry-exporter-otlp", "~> 0.26.3" gem "opentelemetry-instrumentation-all", "~> 0.60.0" ``` ## Step 2: Initialize the instrumentation As per the Ruby on Rails convention, we add an `instrumentation.rb` file to the `config/initializers` directory. ```ruby config/initializers/instrumentation.rb theme={null} require 'opentelemetry/sdk' require 'opentelemetry/instrumentation/all' class ChecklySampler def should_sample?(trace_id:, parent_context:, links:, name:, kind:, attributes:) tracestate = OpenTelemetry::Trace.current_span(parent_context).context.tracestate decision = tracestate.value('checkly') ? OpenTelemetry::SDK::Trace::Samplers::Decision::RECORD_AND_SAMPLE : OpenTelemetry::SDK::Trace::Samplers::Decision::DROP puts(decision) OpenTelemetry::SDK::Trace::Samplers::Result.new(decision: decision, attributes: {}, tracestate: tracestate) end def description 'ChecklySampler' end end OpenTelemetry::SDK.configure do |c| c.use_all() end OpenTelemetry.tracer_provider.sampler = ChecklySampler.new ``` Notice the `ChecklySampler` configuration. This is a custom, **head-based sampler** that will only sample spans that are generated by Checkly by inspecting the trace state. This way you only pay for the egress traffic generated by Checkly and not for any other traffic. Also note that the `use_all()` method will automatically install all available instrumentation libraries. ## Step 3: Start your app with the instrumentation Toggle on **Import Traces** and grab your OTel API key in the **OTel API keys** section of the [Traces page in the Checkly app](https://app.checklyhq.com/settings/account/traces) and take a note of the endpoint for the region you want to use. Checkly OTEL API keys

## What is Distributed Tracing? Distributed tracing tracks requests as they flow through your application's services and infrastructure. It helps you: * Understand request flow across services * Identify performance bottlenecks * Debug failures and errors * Optimize application performance With telemetry traces configured, you will have access to traces in all the places where it matters to more quickly resolve issues: * **Check results:** resolve production outages faster by correlating failing checks with infrastructure traces. * **Test sessions:** understand any failures during test session execution. * **Check editors:** get a live trace while building, editing and debugging check code. ## Getting Started with Checkly Traces To get started with Checkly Traces using OpenTelemetry, pick the scenario that best fits your needs. Instrument your app and send traces directly to Checkly. No need for a 3rd party OTel backend. Send your infrastructure traces to Checkly to get contextualized check failure analysis. Export check results as traces to your 3rd party OTel tooling See this in action in the video below: