AI Bots Read More Than They Refer: What Marketers Should Know

AI crawlers have developed a big appetite. The clicks they send back have not kept pace.

Cloudflare’s 2025 traffic data gives marketers a useful way to think about that gap: the AI crawl-to-referral ratio. It compares how often an AI or search platform requests HTML pages with how often that platform refers a person back to a website.

The takeaway is not to block every bot or chase every new optimization trick. It is to measure machine discovery and human response as two related, but different, outcomes.

The numbers are a little wild

In its 2025 Radar Year in Review, Cloudflare reported that AI bots generated an average of 4.2% of HTML requests across its customer base. Googlebot alone accounted for 4.5%.

The gap between crawling and referrals varied dramatically by platform. Anthropic’s ratio settled mostly between roughly 25,000:1 and 100,000:1 after May. OpenAI peaked near 3,700:1 in March. Perplexity finished much lower, generally below 200:1 from September onward.

One important caveat: native apps may not always pass referral information, so some AI-driven visits can be undercounted. Cloudflare also describes this ratio as volatile. It is a directional signal, not a tidy scoreboard.

21×

User-action crawler traffic grew more than 21 times from January through early December 2025. That is crawling tied more closely to a person’s request, rather than background model training.

What might the crawl-to-referral gap mean?

1. Visibility may happen before the click

An AI answer can mention a company, summarize its product, or quote its research without producing a conventional website session. That makes citations, accurate representation, and share of answer useful companions to rankings and traffic.

2. Referred visitors may arrive with stronger intent

This is a theory worth testing, not a universal rule. If an answer engine handles early research, a person who clicks through may be closer to comparing, validating, or buying. Smaller traffic numbers could still carry meaningful business value.

3. Original information becomes more valuable

Generic summaries are easy to reproduce. First-party research, product details, expert explanations, clear policies, and current documentation give both people and answer engines a reason to use a specific source.

The useful question is not, “Did a bot visit?” It is, “Did our content become the clearest source for the next human decision?”

Retail and software may feel the shift first

Cloudflare found that retail and computer software together attracted just over 40% of AI crawler activity in October 2025. That concentration suggests a few practical possibilities.

• Retail: Product specifications, availability, shipping rules, comparisons, and returns information should be accurate, crawlable, and consistent across pages and feeds.
• Software: Documentation, pricing, integrations, feature definitions, and release notes may influence answers long before a demo request.
• Publishers: The value exchange between crawling, attribution, licensing, and referral traffic will remain a live business question.
• Health and financial services: High-stakes content benefits from named reviewers, clear sourcing, visible dates, and careful update practices.

A practical SEO, AEO, and GEO checklist

Google’s official guidance for generative AI features says that foundational SEO still matters. Google recommends crawlable pages, sound technical structure, useful original content, and a good page experience. It also says no special AI markup or llms.txt file is required for visibility in Google’s generative search features.

Microsoft’s Bing Webmaster guidance adds a helpful content layer: use clear headings, tables, and FAQ sections; support claims with evidence; reduce ambiguity; and keep important information current.

1. Stay discoverable. Check indexing, robots directives, canonical URLs, sitemaps, and JavaScript rendering.
2. Answer the real question early. Lead each important section with a direct, useful response, then add context.
3. Make claims easy to verify. Link to primary sources, name methodologies, show dates, and distinguish facts from theories.
4. Use structured data honestly. Add supported schema when it accurately describes visible page content. Validate it before publishing.
5. Keep brand facts consistent. Product names, pricing, locations, leadership, and policies should agree across owned channels.
6. Measure both sides. Track organic traffic and conversions alongside citations, cited pages, referral sources, and brand accuracy in AI answers.

The short version

AI systems are reading a great deal of the web, and referral traffic tells only part of the story. Marketers do not need to abandon SEO or redesign every page for a mysterious machine audience. The sensible move is to publish distinctive, well-structured information, make it easy to verify, and measure whether search and AI experiences represent the brand accurately.

In other words: keep earning the click, but start paying attention to the answer that comes before it.