Crawl Budget
Crawl budget is the amount of crawling Google is willing and able to spend on a site in a given time, combining how much it wants to crawl with how much the site can be crawled efficiently.
Key takeaways
- Crawl budget = crawl rate (server capacity) + crawl demand (Google's interest)
- Wasted crawling on duplicates, errors, or redirects reduces discovery of important pages
- Optimization is critical for large or frequently changing sites
- Use Google Search Console Crawl Stats report to measure and monitor
- Fix duplicates, soft 404s, and redirect chains before other optimizations
Crawl budget determines how efficiently Google discovers and indexes your content.
Example
An e-commerce site with 50,000 product pages, 10,000 category pages, and 5,000 blog posts. Googlebot has a limited crawl budget per day. If the site has 20,000 duplicate URLs (e.g., session IDs, filter parameters) and 500 soft 404s, Google wastes budget on those instead of crawling new products or updated blog posts. Fixing duplicates and soft 404s can increase the crawl rate for valuable pages.
When Should I Use This?
Crawl budget optimization is most important when:
- Your site has more than 10,000 pages
- Your site is updated frequently (e.g., news, e-commerce)
- You notice important pages are not indexed
- Google Search Console shows high crawl errors or low crawl rate
- You have many duplicate URLs, redirect chains, or soft 404s
What It Is Not
- A direct ranking factor — crawl budget affects discovery, not ranking directly
- A problem for small sites (under 10k pages) — Google usually crawls them fully
- The same as indexing — crawling is fetching URLs; indexing is storing them
- Something you can increase by paying or submitting more sitemaps
Quick Start
- Check Crawl Stats in Google Search Console → Settings → Crawl Stats
- Identify wasted URLs — look for duplicates, soft 404s, redirect chains, and low-value pages
- Fix duplicates — use canonical tags, consolidate similar pages, or block with robots.txt
- Update XML sitemaps — include only high-value, canonical URLs; use
<lastmod>for freshness - Improve server response — faster pages allow Google to crawl more in the same time
Common Mistakes
- Treating crawl budget as a direct ranking factor instead of an efficiency and discovery issue
- Focusing on crawling fixes before removing duplicate URLs, soft 404s, and redirect chains
- Leaving XML sitemaps outdated or filled with low-value URLs
- Blocking the wrong URLs with robots.txt when the real issue is poor internal linking or duplicate content
Next step
FAQ
What is crawl budget?
Crawl budget is the number of URLs Googlebot will crawl on your site within a given time. It combines crawl rate (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl your site based on content quality and popularity).
How to check crawl budget?
You can check crawl budget using Google Search Console's Crawl Stats report, which shows total crawl requests, response times, and errors. Log file analysis also provides detailed crawl patterns.
Does crawl budget affect SEO?
Yes, but primarily for large sites (over 10,000 pages) or sites with frequent updates. For small sites, crawl budget is rarely a limiting factor.
Related topics
Sources
- Google Search Central: Crawl budget management — Primary source for Google’s definition, causes of wasted crawl budget, and optimization guidance.
- Google Search Central: Manage crawling and indexing of site URLs — Useful supporting Google documentation on controlling crawl and index behavior through technical SEO signals.
- Google Search Central: Crawl stats report — Authoritative reference for measuring crawl activity in Google Search Console.
Reviewed by Lucía Marín, Founding editor.