AI Search Crawlers vs. Training Crawlers: A B2B SaaS Access Policy

Do not make one blanket “allow AI” or “block AI” decision for a B2B SaaS website. Separate search discovery, model training, user-triggered retrieval, traditional search indexing, and private access. Then decide which content classes each purpose may reach.

A public product page, technical documentation, a licensed research report, a customer portal, and a staging environment do not have the same discovery value or disclosure risk. A single User-agent: * rule ignores that difference. It can also block the exact search crawler the marketing team wants while failing to protect content that should have required authentication in the first place.

I co-founded Stackmatix, which works on SEO, AI-search visibility, and paid growth. That commercial relationship informs the editorial position here. The policy and worked example are recommendations with hypothetical data, not client results, legal advice, or a promise of inclusion in any search or AI answer.

First, separate five kinds of access

“AI crawler” is too broad to be a useful policy label. Start with the product purpose.

These purposes can use different agents and controls. They can also share infrastructure. Read the current platform documentation before copying a rule from an old article or another company’s file.

How do the current platform controls differ?

The comparison below reflects official documentation checked on October 23, 2026.

OpenAI’s crawler documentation explicitly separates OAI-SearchBot, GPTBot, and ChatGPT-User. It says a publisher can allow the search crawler while disallowing the training crawler, and notes that search systems may take about 24 hours to adjust after a robots.txt change.

Anthropic’s site-owner crawler guidance similarly documents Claude-SearchBot, ClaudeBot, and Claude-User for search, model development, and user-directed access.

Google’s architecture is different. Its common-crawler documentation says Google-Extended is a control token rather than a separate HTTP user agent. The setting covers use for future Gemini model training and certain grounding in Gemini Apps and Vertex AI, but it does not affect inclusion or ranking in Google Search. That means “allow search, block training” is not a perfectly portable policy across providers.

Microsoft’s AI Performance documentation for Bing Webmaster Tools says its reporting covers citations across supported AI experiences and that Bing respects publisher preferences expressed through robots.txt and other supported controls. It also makes a useful measurement distinction: citation counts do not establish ranking, authority, or the role of a page within an answer.

Build the policy by content class

The right unit of governance is not “the website.” It is a content class with an owner, purpose, and access requirement.

Public product and use-case pages

These pages exist to be discovered and evaluated. A company pursuing search visibility will usually want conventional search and selected AI-search agents to reach them. Training access remains a separate policy decision.

The page still needs accurate claims, stable URLs, and useful evidence. Crawler access is eligibility, not distribution. The marketing-claims sourcing checklist helps ensure that a discoverable answer is also defensible.

Public editorial content

Articles designed to explain customer decisions often have high discovery value. Allow the search agents that serve the intended channels if company policy permits it. Decide training access separately and document who owns that decision.

Keep external evidence near the claim and distinguish public facts from hypothetical examples. The SEO, AEO, and GEO guide covers the broader publication and measurement system.

Public technical documentation

Documentation can be valuable in both search and user-triggered assistance because readers need current product behavior. The stronger risk is usually stale or conflicting truth, not discovery itself.

Give technical facts a maintained home and link commercial or editorial pages to it. The canonical-answer routing framework helps determine whether a fact belongs in docs, a landing page, an integration page, or an article.

Licensed, embargoed, or strategically restricted material

Do not assume a crawler directive resolves every contractual or policy question. Establish which uses the company has the right to authorize, which agents the provider documents, and whether the content should be public at all.

If a report is public but training use is restricted by policy, apply the relevant documented training controls and preserve the decision record. If access is limited to licensed readers, enforce authentication and authorization at the origin.

Customer, employee, admin, and staging surfaces

Private content should be private. Require authentication, enforce authorization, prevent public linking and sitemap inclusion, and return the correct status to unauthorized requests. robots.txt is a crawl preference published to everyone; it is not an access-control system.

The same rule applies to sensitive exports, unannounced product pages, customer-specific documents, internal search results, and test environments. If disclosure would matter, protect the resource rather than asking named bots not to fetch it.

Use this access decision matrix

For each content class, make six decisions:

1. Reader: Who is supposed to use the material?
2. Public state: Is it intentionally public, licensed, or private?
3. Discovery value: Which conventional and AI-search surfaces help that reader find it?
4. Training position: Which documented training uses does company policy permit?
5. User-triggered behavior: Should an assistant acting for an allowed user be able to retrieve or interact with it?
6. Enforcement: Is robots.txt sufficient for the preference, or is authentication, an HTTP status, a meta rule, a WAF rule, or another control required?

Write the answers before editing the file. This prevents a developer from being asked to translate “block AI” into infrastructure without knowing what outcome the business actually wants.

Work through a hypothetical SaaS site

Consider a fictional workflow company with five content classes. Every decision below is hypothetical, not a universal recommendation.

The team’s earlier file used User-agent: * and Allow: / for every public hostname. That did not express the training policy, and it left staging discoverable because the environment relied on an obscure URL rather than access control.

The repaired design does two different jobs:

• on the public marketing host, it keeps general search and named search agents available while adding documented provider-specific training preferences;
• on licensed, customer, and staging surfaces, it uses real access controls rather than depending on polite crawler behavior.

The result is not “more optimized.” It is more explicit. Marketing can explain what remains discoverable, engineering can explain what is enforced, and legal or policy owners can approve the exact use being allowed or declined.

What would an illustrative robots.txt policy contain?

Suppose the fictional company chooses to allow OpenAI and Anthropic search discovery, disallow their named training crawlers, preserve Google Search, and disallow Google-Extended after accepting that this also affects the Gemini uses covered by Google’s current documentation.

Its policy would include distinct groups for:

• OAI-SearchBot with public paths allowed;
• GPTBot with the selected paths disallowed;
• Claude-SearchBot with public paths allowed;
• ClaudeBot with the selected paths disallowed;
• Googlebot with public paths allowed;
• Google-Extended with the selected paths disallowed;
• a general group for other agents, plus the canonical sitemap location.

That is a policy outline, not a block to paste blindly. Rule precedence, path matching, provider changes, existing directives, and CDN behavior need to be tested on the actual host. Recheck the linked primary documentation when implementing it.

Do not use robots.txt as a universal deindexing tool. Google’s noindex documentation says the crawler must be allowed to access a page to see its noindex meta rule or HTTP header. A page blocked from crawling can still have its URL discovered elsewhere. Sensitive material should be access controlled; a public page that should not appear in supported search indexes needs the appropriate indexing control.

Test the decision through the whole delivery path

Editing the file is only the first half of the work.

1. Save the previous file and the approved policy version.
2. Test representative allowed and disallowed URLs against the final parser behavior.
3. Confirm the production hostname serves the intended file with a successful response.
4. Inspect redirects so agents do not land on a different host with a different policy.
5. Check CDN, WAF, bot-management, rate-limit, and geo rules. An allowed agent can still be blocked before it reaches the page.
6. Verify legitimate agents using the provider’s current IP or authentication guidance where available; user-agent strings can be spoofed.
7. Inspect access logs for the named agents, requested paths, responses, and unexpected volume.
8. Recheck sitemaps, canonical URLs, noindex rules, and authentication boundaries.
9. Allow for documented processing delay before interpreting a visibility change.
10. Record the verification date, evidence, unresolved limits, and owner.

OpenAI publishes IP ranges alongside its crawler definitions. Google warns that HTTP user-agent strings can be spoofed and documents crawler verification. The practical lesson is consistent: a line in robots.txt does not prove that the intended agent reached the intended page.

Diagnose the failure before changing content

Use the AI citation diagnostic after access is verified. It separates discovery, relevance, original evidence, and the limits of prompt-based observation. Use the AI-search measurement framework to keep citations from being reported as pipeline.

Copy this crawler-access policy record

The durable rule is simple: use crawler controls to express crawler preferences and access controls to protect access. Make the decision by content class and product purpose. Preserve conventional search intentionally. Treat AI search, training, and user-triggered retrieval as separate where the provider allows it—and record the cases where one control still covers more than one use.

Matt Pru

Co-founder and CEO of Stackmatix, startup advisor, and former Head of Sales at MightyHive. Connect with Matt on LinkedIn ↗ · More about Matt →