Robots.txt

What Is Robots.txt?

A misconfigured robots.txt can ban your entire website from Google’s index — a mistake that happens more often than you might think, especially after relaunches. At the same time, a well-thought-out robots.txt helps direct Google’s crawl budget to your most important pages instead of wasting it on admin areas or internal searches. In the AI era, your robots.txt also determines whether AI crawlers like GPTBot or PerplexityBot are allowed to capture your content.

Robots.txt is a text file placed in the root directory of a website (e.g., www.example.com/robots.txt) that gives search engine bots instructions on which areas of the website may be crawled and which may not. This is one of the most important technical SEO tools for controlling crawler behavior. With robots.txt, you can instruct the Googlebot, Bingbot, and other crawlers to avoid certain directories or file types — for example, to save crawl budget or prevent internal search or admin areas from being crawled.

Technically, robots.txt is a simple text file with a clear format: “User-agent: Googlebot” tells Google that the following instructions apply only to the Googlebot. “Disallow: /admin/” prohibits crawling of the /admin/ directory. “Crawl-delay: 5” instructs the bot to wait 5 seconds between two requests. It is important to understand that robots.txt should not be used to protect sensitive content — the file is publicly readable and hides nothing from malicious actors. For real security, you should use noindex or HTTPS authentication.

When creating and managing, website owners should be careful. A mistake in robots.txt can cause important pages to no longer be crawled and disappear from the Google index. Best practice is to only block areas that are truly uncrawlable or unimportant. Most websites should not block: their homepage, important content pages, and the XML sitemap file. Regular testing via Google Search Console shows if robots.txt is causing problems — Google warns when there are parsing issues.