We scanned 22,324 small business websites. Over half block at least one AI crawler.

Between June 15 and August 9, 2026, we tested the websites of 22,324 small businesses across 97 cities and 30 industries to see whether AI crawlers, the programs that let ChatGPT, Claude, and Perplexity learn about businesses, could actually read them. At least half the sites block one or more of these crawlers. Nearly 1 in 5 blocks a crawler that AI search platforms directly depend on. Almost none of the owners chose this.

Key findings

• At least 50.7% of scanned sites (11,315 of 22,324) block one or more AI associated crawlers. This is a floor, not an estimate. Our pipeline only counted a site as a blocker when the block was confirmed, and sites without a findable contact email were left out of the tally even when they blocked crawlers. The true rate is higher.
• 18.7% (4,182 sites) block at least one of the eight key crawlers behind AI search answers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, or Google-Extended. These businesses are partially or fully invisible to the platforms their customers increasingly ask for recommendations.
• 13.1% of all scanned sites block ClaudeBot and 10.9% block GPTBot. Among confirmed blockers, roughly 1 in 4 blocks at least one OpenAI crawler and 28.4% block at least one Anthropic crawler.
• WordPress dominates the problem: 74% of sites blocking key AI search crawlers (3,099 of 4,182) run on WordPress. Security plugins and firewall defaults are the likely cause, since these blocks correlate with platform, not with anything the owners were doing intentionally.
• The remaining 63% of blockers only block secondary bots such as Bytespider (TikTok's scraper, blocked by 69.7% of blockers) or Amazonbot (43.8%). Blocking those does not hurt AI search visibility, which is exactly the point: which bot you block matters more than whether you block.

Blocking rates by crawler

Share of all 22,324 scanned sites confirmed to block each crawler. Every figure is a floor for the reason described in the methodology.

Which industries are most affected

Counts of sites blocking at least one key AI search crawler, by industry. The pattern skews toward high value service businesses, exactly the categories people ask AI platforms to recommend.

Why so many sites block crawlers nobody meant to block

Three mechanisms explain most of what we found:

1. One toggle defaults. Some CDN and security products add AI crawler blocks to robots.txt with a single setting, sometimes enabled by default, without changing any file the owner ever looks at.
2. Security plugins. Bot protection features in popular WordPress plugins treat AI crawlers as threats and return errors to them, while the site looks perfectly normal in a browser.
3. Copied robots.txt files. Blocklists get copied between sites and templates. A rule someone added deliberately in 2023 now travels with themes and starter files into sites whose owners have never read it.

We are not in a position to be smug about this. While preparing this study we found a CDN managed block sitting on top of our own robots.txt, silently overriding the file we had written. If it can happen to a GEO agency, it can happen to anyone, which is rather the point of publishing the data.

Methodology

Websites were sourced from public business listings across 97 cities and 30 service industries, including interior designers, pilates studios, attorneys, financial advisors, wedding planners, and event venues. Each site was tested between June 15 and August 9, 2026 against 15 crawler user agents using three independent signals:

• robots.txt rules: explicit Disallow directives naming each crawler.
• Page level directives: meta robots tags and X-Robots-Tag headers that exclude all bots.
• Live HTTP responses: a real request per crawler user agent, counting only hard block status codes. Small HTML responses were not treated as blocks, so single page app sites were not falsely flagged.

What a business owner should do with this

Check your own doors before doing anything else. Open yourdomain.com/robots.txt and look for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. If any of them is disallowed and you did not choose that, something chose it for you. Our step by step guide covers the full checklist, and the free audit below runs every check in this study against your site.