Crawl Budget
Crawl budget is the number of pages a search engine bot will crawl on your website within a given timeframe, determined by your site's crawl rate limit and crawl demand.
Crawl budget refers to the resources search engine bots allocate to crawling your website. Google's crawler (Googlebot) doesn't have unlimited resources, so it must prioritize which pages to crawl and how often. Your crawl budget determines how many of your pages get discovered and indexed.
Crawl demand is how much Google wants to crawl your site based on popularity, freshness, and site size. For most small to medium websites (under a few thousand pages), crawl budget isn't a concern. Googlebot can easily crawl the entire site.
In practice
- An e-commerce site with 500,000 product pages discovered that Googlebot was spending 60% of its crawl budget on filtered navigation pages, leaving thousands of product pages unindexed
- A news website implemented proper pagination and canonical tags, reducing wasted crawl on duplicate archive pages and improving the indexing speed of new articles
- A travel site blocked faceted search URLs in robots.txt, redirecting crawl budget to their 10,000 destination pages that actually needed indexing
- A marketplace platform noticed their new product listings weren't appearing in search for weeks; optimizing the XML sitemap and internal linking reduced indexing time to 48 hours
Not sure how this applies to you?
A senior engineer walks your site, your search presence and the way enquiries reach you, then writes down the three things worth fixing first. You keep the document either way.