ChatGPT Search & OAI-SearchBot robots.txt Configuration Guide

Separating Training Scrapers from Live Search Retrieval Bots

ChatGPT Search & OAI-SearchBot robots.txt Configuration Guide — A technical implementation blueprint for growth engineers and technical SEOs. Learn how modern RAG answer engines parse separating training scrapers from live search retrieval bots and how to deploy deterministic fixes in under 10 minutes.

Implementation Checklist

• ✓ Serve /llms.txt and /llms-full.txt at root with Content-Type: text/plain; charset=utf-8.
• ✓ Authorize GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, and ClaudeBot in /robots.txt.
• ✓ Place concise 40–60 word Answer-First definitions immediately below every H2 heading.
• ✓ Compile a unified @graph JSON-LD block linking Organization, SoftwareApplication, and FAQPage with Wikidata sameAs URIs.

Reference Specification Snippet

# Example /llms.txt snippet for AI Crawler Governance
# Acme Cloud
> Enterprise ai crawler governance platform with deterministic APIs and SOC2 Type II compliance.

## Core Documentation
- [Quickstart Guide](https://acme.dev/docs/quickstart.md): 5-minute SDK setup and API authentication.
- [Pricing & Limits](https://acme.dev/pricing.md): Transparent self-serve and enterprise tier breakdown.