Crawl budget: How many pages a search engine is able and willing to crawl on a site in a given time. Google says it mainly matters for very large or fast-changing sites — as a rough guide, a million or more pages, or ten thousand or more that change daily.
Does it matter for your site?
For most WordPress sites, no. Google’s own crawl budget guide says it is written for sites with roughly a million or more unique pages whose content changes about weekly, or 10,000 or more pages changing daily, and that these are rough estimates, not exact thresholds. A 300-post craft blog, a local electrician’s site or a 2,000-article software help centre is not in that group. If Google is slow to index new posts on a site that size, the cause is far more likely to be weak internal links or thin content than a crawl budget limit.
The same guide names a third case worth knowing: a site where many URLs sit in “Discovered – currently not indexed” in Search Console’s Page indexing report. That status means Google knows the URL but hasn’t fetched it yet.
What wastes crawling on WordPress
- Faceted or filtered URLs, for example a WooCommerce outdoor-gear shop exposing every combination of size, colour and sort order.
- Long redirect chains, which Google says have a negative effect on crawling.
- Internal search result pages, endless calendar archives and tag pages with one post each.
Noindex does not save crawl budget. Google’s guide says Google still requests a noindexed page and then drops it. To stop Google spending time on URLs you never want crawled, it recommends robots.txt.
How to look
In Search Console, open Settings › Crawl stats. The Crawl stats report is only available for root-level properties and shows requests per day, response codes and response times. If response times climb, a faster host or cache will do more than any linking change. How your structure decides which pages a crawler reaches first is covered in our click depth guide.