Robots.txt
Robots.txt is a text file placed at the root of a website that tells search engine crawlers which pages or sections of the site they are allowed or not allowed to access.
The robots.txt file is one of the first things search engine bots check when visiting your website. Located at yoursite.com/robots.txt, it contains directives that tell crawlers which parts of your site they can access and which they should avoid. It uses a simple syntax with User-agent (which bot), Disallow (which paths to skip), and Allow (exceptions to disallow rules).
You can also use it to point search engines to your sitemap file. It's important to understand that robots.txt is a polite request, not a security measure. Well-behaved bots like Googlebot respect it, but malicious bots may ignore it entirely.
In practice
- A company's new developer accidentally added Disallow: / to robots.txt, blocking Google from their entire site and causing traffic to drop 90% before anyone noticed
- An e-commerce site blocked their faceted navigation URLs via robots.txt, saving thousands of pages of crawl budget for their actual product pages
- A multi-site WordPress installation used robots.txt to prevent staging site content from being indexed by search engines
- A SaaS platform blocked /app/ and /account/ paths to prevent internal dashboard pages from appearing in search results
Not sure how this applies to you?
A senior engineer walks your site, your search presence and the way enquiries reach you, then writes down the three things worth fixing first. You keep the document either way.