ai.txt Generator

What is ai.txt?

ai.txt is a proposed convention for declaring, in plain text at your site's root, which AI bots and crawlers you allow or disallow — and often why. It's designed specifically around AI model training and content-scraping permissions (bots like GPTBot, Google-Extended, CCBot, ClaudeBot), which is a distinct concern from general web crawling.

It's deliberately separate from robots.txt. robots.txt is the long-established, universally respected mechanism for controlling crawl access path-by-path; ai.txt layers on a human- and policy-readable statement of intent about AI training/use specifically, since not every bot that respects robots.txt directives distinguishes between indexing and training use.

When to use it

Publish one if you want an explicit, citable record of your position on AI training access — useful for publishers, blogs, and businesses that want to permit some bots (e.g. for search visibility) while blocking others (e.g. training scrapers), or that simply want a documented policy for legal/PR purposes. It's also useful if you've already added Disallow rules for AI bots in robots.txt and want a clearer, purpose-specific statement alongside them.

Note that ai.txt has no legal or technical enforcement power beyond what a given bot operator chooses to honor — treat it as a stated policy, not a technical block.

Example

Set Purpose to "We welcome AI search indexing but do not consent to model training on our articles," then add rules: GPTBot → Disallow, Google-Extended → Disallow, and a custom SearchBot → Allow. The generator outputs a commented ai.txt with a User-agent/Allow or Disallow block per bot and your contact line, ready to publish at your domain root.