Your firewall is probably blocking the AI crawlers you want

A web application firewall exists to stop things that look like abuse. A bot manager exists to stop automated traffic that costs you money or steals your content. Both are doing their job when they block an AI crawler, because an AI crawler is structurally indistinguishable from the thing they were bought to stop: it requests thousands of pages, it holds no session, it executes no JavaScript, it accepts no cookies, and it arrives from a datacentre address range.

What changed is that some of that automated traffic became the distribution channel. When the choice was between a scraper and a customer, blocking was free. Now a share of the traffic that behaves like a scraper is the mechanism by which a buyer hears your name in an answer, and the same rule that was free in 2021 has a cost attached in 2026.

The specific mechanism is almost always a managed ruleset rather than a rule somebody wrote. CDN and WAF vendors ship curated bot categories that update on their schedule, and several of them added an AI or generative AI category with a default action of block or challenge. If your configuration inherits the managed defaults, and most do, the block appeared in your estate without a change request, a ticket or a deployment. Nobody decided anything. That is precisely why it is so common.

Managed hosting adds a second layer. Some platforms apply their own edge protection above whatever you have configured, and some marketplace security plugins ship AI crawler blocking as an on-by-default feature described as a privacy improvement. It is entirely possible to have three independent systems each capable of returning the 403, which is why the diagnosis has to be empirical rather than a review of the config.

There is no feedback loop. A person who hits a 403 emails support. A crawler that hits a 403 records a failure, backs off, and comes back less often until it stops coming back at all. Nothing in that sequence produces a notification for you.

The monitoring you already have will not catch it either, for three reasons. Analytics platforms filter bot traffic out by design, so the absence of AI crawler hits looks exactly like the normal filtered state. Uptime monitors request the site as a browser, from an allowlisted address, and report green. Search Console and its equivalents cover the search engine that owns them and say nothing about anyone else's crawler.

The result is a specific and quite common failure state: a well-maintained corporate estate, a competent marketing team, an active content programme, and a complete absence from AI answers that everyone attributes to the content being wrong. It is usually not the content.

We found exactly this on a global consumer brand's estate. A managed WAF was returning 403 to every AI crawler across the whole portfolio while the marketing team believed the sites were open and were commissioning more content to fix a visibility problem that no amount of content could fix. The block had never been requested by anyone. It came in with a ruleset. We found it by requesting the pages as each user agent, which is a test anyone can run and almost nobody does.

The whole diagnostic is a request with a substituted user agent. If you can run curl, you can do this without involving anyone. Start with a single agent against a single page.

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  https://www.example.com/

That prints the status code and nothing else. Run it against your homepage, a service or product page, and something deep in the site, because edge rules are frequently path-scoped and a passing homepage proves very little.

Once one agent works, loop the rest. This version prints a status code per agent and is short enough to paste into a terminal.

for UA in \
  "OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  "ChatGPT-User/1.0; +https://openai.com/bot" \
  "GPTBot/1.1; +https://openai.com/gptbot" \
  "ClaudeBot/1.0; +claudebot@anthropic.com" \
  "Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot" \
  "Claude-User/1.0; +https://www.anthropic.com/claude-user" \
  "PerplexityBot/1.0; +https://perplexity.ai/perplexitybot" \
  "Perplexity-User/1.0; +https://perplexity.ai/perplexity-user" \
  "Googlebot/2.1; +http://www.google.com/bot.html" \
  "Applebot/0.1; +http://www.apple.com/go/applebot" \
  "meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler" \
  "Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot" ; do
  CODE=$(curl -sS -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (compatible; $UA)" https://www.example.com/)
  printf "%s  %s\n" "$CODE" "${UA%%/*}"
done

Two refinements are worth making before you trust the output. First, run it from an address that is not your office, because corporate ranges are often allowlisted and will give you a false pass. Second, at least once, drop the -o /dev/null and read the body, because a challenge page returns a perfectly healthy 200 while containing none of your content.

Half the blocks we find are not at the firewall at all. They are four lines in a text file that nobody has read since the site launched. Robots.txt is enforcement by convention rather than by force, but the major AI operators do honour it, so a disallow there is as effective as a 403 for the crawlers you care about.

Two patterns account for most of it. The blanket disallow, which is the correct configuration for a staging environment and gets copied to production during a migration. And the over-broad wildcard, where a rule written to hide internal search results also hides every URL carrying a query string, which on a parameterised catalogue can be most of the site.

# The blanket disallow. Every crawler, every path. Usually left over from a staging site.
User-agent: *
Disallow: /

# The over-broad wildcard. Intended to hide a search results page, actually hides the catalogue.
User-agent: *
Disallow: /*?

# What a permissive-but-deliberate robots.txt looks like instead.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

Content-Signal: search=yes, ai-input=yes, ai-train=no

Sitemap: https://www.example.com/sitemap.xml

The last block above is what a deliberate configuration looks like: explicit allows for the search and user-triggered agents, an explicit disallow for the training crawler if that is your policy, a content signal expressing the intent behind it, and a sitemap reference. Every line is a decision somebody made.

Two smaller points that cause real problems. Robots.txt is per host and per protocol, so a subdomain and a www variant need their own files and frequently disagree. And a crawler that cannot fetch robots.txt at all, because the firewall blocks that request too, will generally treat the site as disallowed. Test the robots.txt URL with the same user agent substitution you used for the pages.

There is a real problem underneath all of this, and it is not purely technical. Publishers want to be found and quoted, and many of them do not want their archive absorbed into a training corpus. Robots.txt was never designed to express that difference. It has one verb, which is whether a crawler may fetch a path, and it says nothing about what may be done with what was fetched.

Cloudflare's Content Signals Policy is the most developed attempt to close that gap. It adds a machine-readable line to robots.txt that separates three uses: search, meaning inclusion in a search index with a link back; ai-input, meaning use as input to a generated answer; and ai-train, meaning use as training data for a model. Each can be set to yes or no independently, so a publisher can state the common position, which is yes to search, yes to being quoted with attribution, no to training, without having to express it as a crawl block that also removes them from answers.

Be clear-eyed about what it is. It is a declaration, not a control. It has no enforcement mechanism, compliance is voluntary, and adoption across operators is early. What it gives you today is a documented, specific and public statement of intent, which is worth more than it sounds if you are the person who has to explain the organisation's position to a legal team or a rights holder.

The wider point is that access and permission are two different questions and most estates answer both with one badly chosen firewall rule. Getting them apart is the actual work. If you want help doing it, that sits inside our AI search optimisation engagement and, where the policy question is the harder half, our AI consulting practice. The glossary defines the agent names if you need to circulate them internally.