Was ist Crawl Budget?
Kurzdefinition
Das Crawl-Budget ist die Anzahl der Seiten, die ein Suchmaschinen-Bot innerhalb eines bestimmten Zeitraums auf Ihrer Website crawlen wird.
Crawl budget refers to the resources search engine bots allocate to crawling your website. Google's crawler (Googlebot) doesn't have unlimited resources, so it must prioritize which pages to crawl and how often. Your crawl budget determines how many of your pages get discovered and indexed.
Two factors determine crawl budget. Crawl rate limit is the maximum number of simultaneous connections Googlebot will use to crawl your site without overloading your server. Crawl demand is how much Google wants to crawl your site based on popularity, freshness, and site size.
For most small to medium websites (under a few thousand pages), crawl budget isn't a concern. Googlebot can easily crawl the entire site. But for large websites with tens of thousands or millions of pages, like e-commerce stores or news sites, crawl budget becomes critical.
Wasted crawl budget is a common problem. If Googlebot spends time crawling duplicate pages, parameter-heavy URLs, session IDs, internal search results, or other low-value pages, it has less budget available for your important content. Proper use of robots.txt, canonical tags, and noindex directives helps direct crawl budget toward your most valuable pages.
Warum es wichtig ist
If important pages on your site aren't being crawled, they can't be indexed, and if they're not indexed, they can't rank. For large websites, inefficient crawl budget management means some of your best content may never appear in search results.
Crawl budget optimization is especially important after large site migrations, when adding significant amounts of new content, or when you notice that Google is slow to index changes to your site.
Praxisbeispiele
An e-commerce site with 500,000 product pages discovered that Googlebot was spending 60% of its crawl budget on filtered navigation pages, leaving thousands of product pages unindexed
A news website implemented proper pagination and canonical tags, reducing wasted crawl on duplicate archive pages and improving the indexing speed of new articles
A travel site blocked faceted search URLs in robots.txt, redirecting crawl budget to their 10,000 destination pages that actually needed indexing
A marketplace platform noticed their new product listings weren't appearing in search for weeks; optimizing the XML sitemap and internal linking reduced indexing time to 48 hours
Verwandte Begriffe
Technical SEO
Technisches SEO bezeichnet die Optimierung der Website-Infrastruktur, damit Suchmaschinen Ihre Seiten effizient crawlen, indexieren und rendern können.
Schema Markup
Schema-Markup ist strukturierter Datencode, der Ihrer Website hinzugefügt wird und Suchmaschinen hilft, Inhalte zu verstehen und Rich Results zu ermöglichen.
On-Page SEO
On-Page-SEO umfasst die Optimierung einzelner Webseiten — Inhalte, HTML-Code und interne Links — um in Suchmaschinen höher zu ranken.
SERP (Search Engine Results Page)
Eine SERP ist die Seite, die eine Suchmaschine als Antwort auf eine Anfrage anzeigt und organische Ergebnisse, bezahlte Anzeigen und Featured Snippets enthält.
Benötigen Sie Hilfe mit crawl budget?
Unser Team hilft Ihnen, dieses Konzept in die Praxis umzusetzen. Fordern Sie eine kostenlose Beratung an, um Ihr Projekt zu besprechen.