Skip to content
SEO Catalyst

Glossary

What is a web crawler?

A web crawler is a program that search engines use to discover and read web pages by following links from one page to the next. Googlebot is the crawler Google uses.

Why it matters

A page that no crawler reaches cannot be indexed, so it cannot rank. Crawlers find pages through links and through the XML sitemap.

AI companies run crawlers too. Their crawlers read pages so that AI assistants can use the content in answers.

How we handle it

We crawl a site with our own crawler, Screaming Frog, to see what a search engine sees. Server logs then show which pages the real crawlers request. The difference between the two lists tells us where crawling is wasted.

Related service: Technical SEO services
Full guide: Technical SEO checklist

Related terms

Keep reading

All glossary terms

See how your site performs in search

Want this checked on your own site? Send us the address and we will tell you what we would look at first.

Free SEO audit Book a call