Web crawler: A program that discovers pages by fetching them and following the links it finds. Search engines use crawlers to build their index; audit tools use them to map a site.
How crawlers see your site
A crawler starts from addresses it already knows, fetches each one, reads the links in the HTML and adds new ones to its queue. That is why internal links matter so much: Google only follows links written as an <a> element with an href, according to its guide to crawlable links. A “read more” button wired up only in JavaScript, with no href, is invisible to it.
Googlebot is really two crawlers, Googlebot Smartphone and Googlebot Desktop, and because Google indexes the mobile version of most sites, most requests come from the smartphone one. If your theme hides links in the mobile layout, Google mostly sees the site without them. Plenty of other bots claim to be Googlebot, which is why Google says to verify it by reverse DNS or its published IP ranges rather than trusting the user-agent.
WordPress settings that affect crawlers
- Settings › Reading › “Discourage search engines from indexing this site”, which adds a noindex, nofollow robots tag to every page. Handy on a staging site, and a disaster when it is left ticked after launch.
- robots.txt. WordPress generates a virtual one at
/robots.txtwhen no real file exists. It controls crawling, not indexing: Google notes a blocked URL can still appear in results. - Your XML sitemap, which hands crawlers a list of URLs to start from.
Audit crawlers
SEO tools run their own crawlers to map a site the way a search engine would, and they should behave politely: a few requests at a time, slowing down when the server struggles, and obeying robots.txt. Their main value is showing you what a search engine can reach by links alone, which is how you find orphan pages and pages buried too deep. Our guide to measuring and reducing click depth shows how a crawl from the home page reveals which posts a crawler reaches last.