AI crawlers now hit 403 errors on 13.47% of requests. Check your firewall before buying AI visibility

What actually happened

TechnologyChecker’s Q3 2026 analysis of Cloudflare Radar data reports that 13.47% of requests by AI-identified bots received a 403 Forbidden response, compared with 5.98% in Q3 2025. The measurement covers requests across Cloudflare’s observed traffic, not the percentage of individual websites that block AI. September data remains provisional, and Cloudflare does not identify the reason for every denial.

The report also examines robots.txt directives. Those declarations tell cooperative crawlers what a site prefers; they do not determine what happens when a request reaches Cloudflare’s bot controls, a web application firewall, a rate limit or the origin server. An explicitly allowed crawler can still be denied. Search crawlers, training crawlers and user-triggered fetchers also serve different purposes, so a single rule for every AI-branded bot is too crude.

Why a small team cares

Take a local services company that has just rewritten its service pages so an AI assistant can accurately explain its prices and locations. Its SEO specialist checks robots.txt, sees no obvious obstacle and reports the site as ready. Yet a challenge rule might reject the crawler before it can read the new pages. The content work is real, but access is broken.

A useful audit therefore needs evidence from the request path: which verified crawler tried which URL, at what time, what status it received and which rule responded. A generic request with a copied user-agent string is not enough to prove a real crawler is allowed. For a small team, a few relevant URLs and a week of logs give a better first answer than a sprawling AI-ranking dashboard.

Hype vs useful

A 403 does not prove lost sales, reduced citations or even that the blocked request was commercially valuable. Some bot traffic should be rejected. TechnologyChecker’s aggregate number cannot diagnose one domain or distinguish every legitimate request from impersonation. Nor should an AI visibility agency promise that opening the firewall will cause ChatGPT or another service to recommend the company.

The useful decision is operational: verify access first, then measure whether discovery and referrals improve. Keep training opt-outs distinct from search access, avoid a blanket WAF bypass and document changes so an aggressive security update does not quietly reverse them.