robots.txt Generator

What is robots.txt?

robots.txt is a plain-text file at the root of your domain (e.g. https://yoursite.com/robots.txt) that crawlers check before requesting anything else on your site. It uses User-agent blocks to target specific crawlers (Googlebot, Bingbot, or * for all of them) and Disallow/Allow rules to say which paths they may or may not fetch. It's a request, not an access-control mechanism — well-behaved crawlers honor it, but it doesn't block anyone from reaching a URL directly, and it doesn't prevent a page from being indexed if other sites link to it.

When to use it

Use robots.txt to keep crawlers out of admin panels, staging paths, internal search-result pages, and duplicate content generated by filters or session IDs — this keeps your crawl budget focused on pages you actually want indexed instead of wasted on infinite parameter combinations. Add a Sitemap directive pointing to your sitemap.xml so crawlers discover it without you submitting it separately per crawler, and a Host directive if you need to declare a preferred domain (mainly relevant for Yandex).

Don't use Disallow to try to keep a page out of Google's index — if it's linked from elsewhere, Google can still index the URL (without a snippet) even if it never crawls the content. Use a noindex meta tag on the page itself for that.

Example

A typical setup for a SaaS marketing site: User-agent * with Disallow: /admin and Disallow: /app (the logged-in product, which shouldn't be crawled), Allow: / for everything else, plus Sitemap: https://harborlytics.io/sitemap.xml. Add a second User-agent block targeting a specific bot (e.g. an aggressive AI scraper) with its own Disallow rules if you need different treatment per crawler.