Was ist Robots.txt?
Kurzdefinition
Robots.txt ist eine Textdatei, die Suchmaschinen-Bots darüber informiert, welche Seiten oder Bereiche der Website nicht gecrawlt oder indexiert werden sollen.
The robots.txt file is one of the first things search engine bots check when visiting your website. Located at yoursite.com/robots.txt, it contains directives that tell crawlers which parts of your site they can access and which they should avoid. It uses a simple syntax with User-agent (which bot), Disallow (which paths to skip), and Allow (exceptions to disallow rules).
Common uses include blocking search engines from crawling admin areas, staging environments, duplicate content (like print versions of pages), internal search result pages, and private user account areas. You can also use it to point search engines to your sitemap file.
It's important to understand that robots.txt is a polite request, not a security measure. Well-behaved bots like Googlebot respect it, but malicious bots may ignore it entirely. Sensitive content should be protected with authentication, not robots.txt.
Misconfigured robots.txt files are one of the most common technical SEO mistakes. A single misplaced directive can accidentally block your entire site from being indexed, or prevent search engines from accessing CSS and JavaScript files they need to properly render your pages.
Warum es wichtig ist
Robots.txt directly controls what search engines can and cannot see on your website. A well-configured file helps search engines focus their limited crawl budget on your most important pages. A misconfigured one can make your entire website invisible to Google.
For large websites, robots.txt is essential for crawl budget management — preventing bots from wasting time on low-value URLs means they spend more time indexing the pages that matter.
Praxisbeispiele
A company's new developer accidentally added Disallow: / to robots.txt, blocking Google from their entire site and causing traffic to drop 90% before anyone noticed
An e-commerce site blocked their faceted navigation URLs via robots.txt, saving thousands of pages of crawl budget for their actual product pages
A multi-site WordPress installation used robots.txt to prevent staging site content from being indexed by search engines
A SaaS platform blocked /app/ and /account/ paths to prevent internal dashboard pages from appearing in search results
Verwandte Begriffe
Technical SEO
Technisches SEO bezeichnet die Optimierung der Website-Infrastruktur, damit Suchmaschinen Ihre Seiten effizient crawlen, indexieren und rendern können.
Crawl Budget
Das Crawl-Budget ist die Anzahl der Seiten, die ein Suchmaschinen-Bot innerhalb eines bestimmten Zeitraums auf Ihrer Website crawlen wird.
Sitemap
Eine Sitemap ist eine Datei oder Webseite, die alle Seiten einer Website auflistet und Suchmaschinen hilft, Inhalte effizienter zu entdecken und zu indexieren.
Indexing
Indexierung ist der Prozess, bei dem Suchmaschinen Informationen über Webseiten analysieren und in ihrer Datenbank speichern, um sie in Suchergebnissen verfügbar zu machen.
Benötigen Sie Hilfe mit robots.txt?
Unser Team hilft Ihnen, dieses Konzept in die Praxis umzusetzen. Fordern Sie eine kostenlose Beratung an, um Ihr Projekt zu besprechen.