Agentic AI Traffic: The Good and the Bad

Agentic AI traffic is now a normal part of every marketing site’s logs, and treating it all as “bad bots” is as risky as letting all of it through. Some of these visitors train models on your content and never send anyone back; others bring buyers who convert better than your average visitor. This guide explains the three main kinds of AI traffic, how to tell them apart, and a practical allow, limit or block policy you can use on a marketing site.

Three kinds of agentic AI traffic hitting your site

“AI traffic” is not one thing. The big AI vendors now publish separate user agents for separate jobs, and the job matters more than the brand name.

AI training crawlers

These bots collect pages in bulk to train future models. Examples include OpenAI’s GPTBot, Anthropic’s ClaudeBot and Common Crawl’s CCBot. Google uses a separate robots.txt token, Google-Extended, to control whether content it crawls may be used for Gemini training; according to Google’s crawler documentation, Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

Training is by far the largest category. Cloudflare’s crawl-to-click analysis from August 2025 found training accounted for nearly 80% of AI bot crawling, with search at about 17% and user actions at 3.2%.

AI search and retrieval fetchers

These bots index or fetch pages so an AI assistant can cite and link to them. OpenAI’s bot documentation describes OAI-SearchBot as the crawler “used to surface websites in search results in ChatGPT’s search features,” and ChatGPT-User as the agent used for user-initiated actions. Perplexity’s bot documentation draws the same line between PerplexityBot (search indexing, not used for foundation model training) and Perplexity-User (fetches a page when a user asks a question). In February 2026, Anthropic split its crawlers into ClaudeBot, Claude-SearchBot and Claude-User, as Search Engine Journal’s report on Anthropic’s updated crawler docs explains.

Autonomous browsing agents acting for real users

The newest category is agents that drive a real browser to complete a task: compare plans, fill a cart, book a demo. OpenAI’s ChatGPT agent runs in a cloud browser, and when OpenAI launched the ChatGPT Atlas browser in October 2025, its agent mode let paying users ask ChatGPT to complete tasks in the browser on their behalf. These sessions load your JavaScript, fire your tags and click your buttons, much like a person would.

The good: why some AI traffic is worth welcoming

Blocking everything with “AI” in the name can cost you real customers and visibility.

• AI referral traffic converts. Adobe’s January 2026 analysis of holiday shopping data reported that traffic to retail sites from generative AI tools rose 693.4% year over year, and that those visitors converted 31% higher than other traffic, spent 45% more time on site and were 33% less likely to bounce.
• Visibility in AI answers depends on search fetchers. OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers. Anthropic’s documentation warns that blocking Claude-SearchBot or Claude-User “may reduce your site’s visibility” in user searches.
• Agents increasingly represent real buyers. A signed shopping or research agent is often a qualified prospect who has delegated the browsing. Cloudflare describes these as agents “directed by end users” doing things site owners want, “such as planning a trip, ordering food, or making a purchase.”

The good traffic has something in common: it is declared, it respects your rules and it sends people or revenue back to you.

The bad: where agentic AI traffic costs you

The same technology creates real costs and risks, especially when bots hide what they are.

• Heavy crawling with little return. Cloudflare measures a “crawl-to-refer” ratio: how many pages a platform fetches for each visitor it sends back. In Cloudflare’s August 2025 breakdown of AI crawler traffic by purpose, Anthropic’s ratio was nearly 50,000:1, OpenAI’s 887:1 and Perplexity’s 118:1 for the first week of August. Cloudflare notes that referrals from native apps may lack a referrer header, so the ratios can overstate the gap, but the direction is clear.
• Stealth, undeclared crawlers. In Cloudflare’s August 2025 report on Perplexity’s crawling, the company said that when its declared bots were blocked, Perplexity used a generic Chrome-on-macOS user agent and rotated IP addresses and networks (ASNs) to keep fetching content, at 3–6 million requests per day. Cloudflare contrasted this with OpenAI, whose ChatGPT-User fetched robots.txt and stopped when disallowed.
• Agents that look exactly like people. HUMAN Security’s guide to ChatGPT Atlas traffic notes that Atlas’s default user agent “matches Chrome on macOS, with no ‘Atlas’ token,” and advises assuming you “cannot reliably detect Atlas with simple user agent filters.”
• Scraping at scale. Akamai’s November 2025 research on AI bots reported AI bot activity up 300% over the past year, and found that more than 90% of AI bot triggers in healthcare came from scraping.
• Form spam and fake leads. An agent that can fill a checkout can also fill a lead form. Without controls, automated submissions reach your CRM and your sales team.
• Analytics and attribution pollution. Agents that run JavaScript fire pageviews, events and conversion pixels. That inflates sessions, skews conversion rates and can teach ad platform algorithms to optimize for the wrong audience. We cover the paid media side in our guide to agentic AI risks for paid media.

How to identify AI bots and agents

No single signal is enough. Layer them, from weakest to strongest.

User agents and robots.txt

Declared bots announce themselves in the User-Agent header, and robots.txt lets you set rules per bot. This works for well-behaved crawlers, but user agents are trivially spoofed, and robots.txt is a request, not an enforcement mechanism. Note also that the vendors treat user-initiated fetchers differently: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores robots.txt rules,” while Anthropic says all three of its bots respect robots.txt.

Published IP ranges

Most major vendors publish IP lists for their bots. OpenAI publishes separate files for GPTBot, OAI-SearchBot and ChatGPT-User, and Perplexity publishes lists for PerplexityBot and Perplexity-User. Checking that a request claiming to be GPTBot actually comes from GPTBot’s published ranges filters out most impostors. The limitation, as Cloudflare points out, is that IP ranges change over time and can be shared by many users of the same infrastructure.

Web Bot Auth and HTTP message signatures

The strongest signal is cryptographic. Cloudflare’s Web Bot Auth proposal from May 2025 uses HTTP Message Signatures (RFC 9421) so a bot or agent signs each request with the Signature, Signature-Input and Signature-Agent headers, and the site verifies it against a published public key. OpenAI’s ChatGPT agent allowlisting guide confirms its agent signs requests this way, with a Signature-Agent value of “https://chatgpt.com”. In August 2025, Cloudflare introduced a signed agents category covering ChatGPT agent, Block’s Goose, Browserbase and others, so site owners can allow them without trusting every Chrome-shaped visitor.

Network-level controls and pay per crawl

Infrastructure providers now make AI policy a setting. On July 1, 2025, Cloudflare announced that new domains would block AI crawlers by default, noting that over one million customers had already turned on its one-click AI crawler block. The same day it launched pay per crawl in private beta, which lets publishers allow, charge or block each crawler, using the HTTP 402 “Payment Required” status code.

A note on llms.txt

llms.txt, proposed by Jeremy Howard of Answer.AI in September 2024, is a Markdown file that summarizes your site for language models. It is a content hint, not an access control. Google’s John Mueller said in 2025, as PPC Land reported on llms.txt adoption, that “none of the AI services have said they’re using LLMs.TXT.” Adding one is harmless, but it will not stop or identify a single bot.

An allow, limit, block policy for marketing sites

Most marketing sites want to be found in AI answers, welcome real buyers who use agents, and keep training scrapers and impostors off lead forms and ad data. A practical framework:

1. Allow verified search and retrieval bots. Let OAI-SearchBot, Claude-SearchBot, PerplexityBot and similar declared search crawlers reach public pages, verified against their published IP ranges.
2. Allow signed agents, but watch what they do. Treat requests with valid Web Bot Auth signatures as legitimate visitors acting for a person. Let them browse and buy, and label their sessions separately in reporting.
3. Decide on training crawlers deliberately. Blocking GPTBot, ClaudeBot, CCBot and Google-Extended does not remove you from Google Search or AI search results. Allow them only if being in future training data is worth the crawl cost to you.
4. Limit anything unverified that claims to be an AI bot. A request using a known AI user agent from outside that vendor’s IP ranges is an impostor. Rate-limit or block it.
5. Protect conversion points. Keep lead forms, demo requests, checkout and gated content behind bot checks regardless of what the visitor claims to be.
6. Keep agents out of your ad signals. Exclude AI agent sessions from the conversions you send back to Google, Meta and other ad platforms so bidding algorithms learn from humans.
7. Block stealth crawlers. Anything with a browser user agent that behaves like a crawler, rotates networks or ignores robots.txt should be blocked at the network edge.
8. Review monthly. Vendors add and rename bots often. Recheck your robots.txt, allowlists and AI traffic share at least once a month.

Where Ðeny fits

Most of this policy depends on seeing AI traffic clearly in the first place. Ðeny Intelligence Hub classifies every visit as Clean, Good Bot, Residential Proxy, invalid traffic (IVT), Out-of-Geo or AI Agent, and rolls it up into a Traffic Health Score from A to F with AI summaries. UTM Breakdown shows how much AI agent and bot traffic each channel is sending, so you can see which campaigns are paying for machines instead of people.

Ðeny Bot Shield blocks unwanted traffic in real time, and persistent UTM attribution keeps your reporting tied to real visitors. It deploys with a one-line script. Learn more on the Ðeny features overview.

The bottom line on agentic AI traffic

Agentic AI traffic is neither friend nor foe by default. Declared search bots and signed agents can bring visibility and high-intent buyers; undeclared crawlers and human-mimicking agents drain resources and corrupt the data your marketing depends on. The winning approach is to verify first, then allow, limit or block by purpose, and keep agent sessions out of your attribution and ad signals. For more on keeping attribution clean, read our guide to marketing attribution.

Want to see how much of your traffic is AI agents, and what it is costing you? Request a Ðeny demo.