The Current State of Content Negotiation for AI Agents (Feb 2026)

The web was built for humans, but now the agents are taking over.

Humans look at a web page and see content rendered by their browser. AI agents see 180,000 tokens of nav bars, footers, and div soup — burning through their context window on junk that makes them slower and stupider.

The web needs to evolve, and we as developers are driving the shift. AI agents like Claude Code, Cursor, Codex, and Gemini are how we interact with documentation, CLIs, and products today. Every time the robots fetch a page, they parse markup that adds no value.

Cloudflare and Vercel recently published posts about solving the context waste problem with good old content negotiation, and the Accept header.

Both posts explain the concept well, but they're missing a crucial piece: which agents currently send these headers? And what's the best way to support the agentic web?

Let's fill those gaps.

The Problem: HTML Is Expensive and not Agent-friendly

When an AI agent fetches a URL, it doesn't see what you see. It sees this:

<div class="wrapper">
  <nav class="site-nav" aria-label="Main">
    <ul class="nav-list">
      <li class="nav-item"><a href="/docs">Docs</a></li>
      <!-- 47 more nav items -->
    </ul>
  </nav>
  <main class="content-wrapper layout-default">
    <article class="docs-content" id="main-content">
      <h1 class="title heading-1">Getting Started</h1>
      <!-- finally, some actual content -->

All those <div> wrappers, class names, ARIA labels, and semantic markup have zero value for an LLM. But every character counts against its context window.

The numbers speak for themselves:

• Cloudflare measured an 80% token reduction when serving markdown instead of HTML
• Vercel saw page content drop from 500KB to 2KB — a 99.6% reduction

The web wasn't always this heavy. But today, actual content only makes up a fraction of what gets sent over the wire. Users with limited mobile data feel it. So do people on slow connections. And now agents, paying by the token, feel it too.

The Solution: Content Negotiation

HTTP has had a mechanism for this since HTTP/1.1. It's called content negotiation, and it works through the Accept request header.

When a client makes a request, it can say, "Hey! I'd prefer markdown, but I'll take HTML if that's all you've got.":

Accept: text/markdown, text/html

If the server supports it, it responds with clean Markdown. If not, the client receives the HTML and handles it. This approach is simple, backwards-compatible, and elegant. And it's been part of HTTP since forever.

Here's what that approach looks like in practice, using curl:

# Request the Checkly docs with a proper `Accept` header
curl <https://www.checklyhq.com/docs/concepts/monitoring-as-code/> \\
  -H "Accept: text/markdown"

If you request the Checkly docs with a proper Accept: text/markdown header, you get structured markdown that an LLM can actually use instead of a massive HTML document.

That's a 99.6% reduction in size and 99.7% fewer tokens — independently measured on our own docs, and strikingly close to Vercel's numbers. And the best thing: it's the same content, but at a fraction of the cost.

We at Checkly believe agents are our next customers. When someone asks their agent, "How do I set up a Playwright Check Suite with Checkly?" we want them to receive clean, structured context, not a soup of HTML tags that burn through their context window. Our docs, of course, are at the center of supporting the agentic web.

What Agents Are Actually Sending The Required Accept Header?

Here's the one-million-dollar question that Cloudflare and Vercel didn't cover in their posts: which agents use content negotiation today?

To answer this question, we let common agents request the URL https://httpbin.org/headers with their native webfetch tools. The page responds with the received request headers in JSON format. Easy to parse and analyze.

❯ Can you fetch <https://httpbin.org/headers>. What are the displayed headers?

⏺ Fetch(<https://httpbin.org/headers>)
  ⎿  Received 254 bytes (200 OK)

Here are the request headers each agent sends (Accept and User-Agent for fun):

✅ "Fetch" in Claude Code (v2.1.38) requests Markdown

Accept: text/markdown, text/html, */*

User-Agent: axios/1.8.4

✅ "WebFetch" in Cursor (2.4.28) requests Markdown

Accept: text/markdown,text/html;q=0.9,application/xhtml+xml;q=0.8,application/xml;q=0.7,image/webp;q=0.6,*/*;q=0.5

User-Agent: Mozilla/5.0 ... Chrome/139.0.0.0 Safari/537.36

✅ "webfetch" in OpenCode (1.2.5) requests Markdown

Accept: text/markdown;q=1.0, text/x-markdown;q=0.9, text/plain;q=0.8, text/html;q=0.7, */*;q=0.1

User-Agent: Mozilla/5.0 ... Chrome/139.0.0.0 Safari/537.36

❌ "unknown tool" in OpenAI Codex 260203.1501 (523) doesn't request Markdown

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9

User-Agent: Mozilla/5.0 ... ChatGPT-User/1.0; +https://openai.com/bot

❌ "WebFetch" in Gemini CLI (0.28.2) doesn't request Markdown

Accept: */*

User-Agent: GoogleAgent-URLContext

❌ "fetch_webpage" in GitHub Copilot (GPT-5.2-Codex) doesn't request Markdown

Accept: text/html,application/xhtml+xml, ...

User-Agent: Mozilla/5.0 ... Code/1.109.3 ... Electron/39.3.0

❌ "read_url_content" in Windsurf (1.9552.21) doesn't request Markdown

Accept: */*

User-Agent: colly - <https://github.com/gocolly/colly>

Key findings

• Only 3 out of 7 agents request markdown — Claude Code, Cursor, and OpenCode are the only ones asking for text/markdown.
• Gemini CLI and Windsurf don't negotiate at all — */* means "give me anything." No markdown preference, no optimization.
• OpenCode and Cursor use the q factor — they set explicit quality factors (e.g., text/markdown;q=1.0, text/html;q=0.7) to prioritize markdown over HTML. The q parameter is HTTP's way of expressing preference as a weight between 0 and 1. Claude Code skips q values and relies on the header value order instead.

The agents requesting markdown are already giving their users an advantage. What about the ones that are not, though? They're burning tokens on HTML that could have been avoided. We don't know why they haven't adopted content negotiation yet, but we expect them to follow suit soon. The benefits are too obvious to ignore.

How do agents actually process fetched content?

We can't know exactly what closed-source agents do under the hood, but from what's publicly documented Claude Code uses a two-stage architecture: the main conversation calls WebFetch, the response is converted to markdown, and a smaller, faster model summarizes it before injecting it into your conversation.

This agent-side optimization is impressive — yet completely unnecessary. Converting HTML to markdown, truncating content, and spinning up secondary models to summarize the content is a lot of work to retrieve available online content. It's on us, the people running servers, creating products, and building the web, to meet agents where they are. Content negotiation has been in HTTP since 1997. When someone requests markdown, we all need to "just respond with markdown".

We at Checkly believe text/markdown will become the standard for agent-to-server communication. The direction is clear, even if not all agent tools are yet on board. If your docs don't respect the Accept header, you're forcing agents to parse HTML, resulting in burned tokens, slow responses, and degraded-quality answers.

Beyond Headers: The Full Picture

Content negotiation is an important step toward improving agentic workflows, but it's not the end goal. Ideally, agents shouldn't need to search the web at all. The right context should be provided upfront, either by you as the agent user or by us as the developer tooling provider.

There are three emerging patterns that complete the picture:

llms.txt

Another approach gaining traction is the use of llms.txt files. These file points agents to (markdown) versions of key pages, similar to how robots.txt guides crawlers. It's a good complement to content negotiation: the Accept header optimizes how content is served, while llms.txt helps agents discover what content exists in the first place.

We at Checkly opted for serving our doc’s key pages on checklyhq.com/llms.txt because it’s most likely that agents want to set up or control the monitoring infrastructure, or help a user using the platform.

# Checkly Docs

## Docs

- [Changing your email or password in Checkly](https://checklyhq.com/docs/admin/changing-your-email-password.md): Learn how to change your email address or password in your Checkly account
- [Creating an API key in Checkly](https://checklyhq.com/docs/admin/creating-api-key.md): Learn how to create and manage user and service API keys for the Checkly API and CLI
- ...

Agent skills

Skills are packaged, agent-readable instructions that tell an AI agent how to use a specific tool or product — no web search required. They go a step further than content negotiation: in addition to publishing our documentation online, we also released a first agent skill to provide executable guidance.

Install the Checkly monitoring skill with:

npx skills add checkly/checkly-cli

Once installed, your favorite agent knows how to create checks, configure alerts, and deploy monitoring infrastructure — not because it read our docs or searched the web, but because we gave it structured and up-to-date instructions designed for the new world of AI agents.

Agent-friendly Developer Platforms

Content negotiation with the Accept header, llms.txt and skills are the obvious choices to improve the agent experience today, but we're already working on the next iteration.

Here's what's coming to the Checkly Platform and Monitoring as Code:

Markdown-first CLI output

AI agents love CLIs, and we're improving the output of CLI commands like checkly test and checkly deploy to be structured and parseable. When an AI agent runs your checks, it should get clean, context-efficient results — not output designed for human eyes scanning a terminal. If the agent is facing issues, the CLI should provide context to help it get unstuck.

We're also exploring a new checkly skills command that will provide all the required context and documentation inside the CLI itself to get your agent back on track.

Markdown-first Core Features

Checkly's monitoring platform is designed to deliver critical information when your infrastructure experiences issues or outages. So far, you, as a human, needed to know what's broken in your infrastructure and since when. This will change — very soon, it won't be you asking for critical information about your platform's health; it will be your agent, supervised by you.

As a result, our API responses and platform core features will become token-efficient and agent-friendly. Watch the space! We've got many things in the works to make "Hey Agent, I received an alert. Can you fix it?" a reality.

Wrapping Up

The web has a new class of visitors. They don't have eyes, they pay by the token, and they're how developers discover and interact with your product.

Not every agent requests markdown today, but the ones that do are already giving their users a better experience — faster responses, lower costs, higher quality answers. The direction is clear.

Here's what you can do right now:

1. Using agents? — test what headers your agent tools send. Some agents optimize for you. Others don't. Know which one you're using. Consider a switch to save tokens.
2. Building for the web? — support Accept: text/markdown. Agents are your next users. Serve them well, or watch them struggle with your docs and product.

If you're monitoring your sites, apps, and infrastructure while riding the agentic train, check out Checkly and Monitoring as Code. It's AI-ready by design. Get started with AI-powered monitoring.