Glossary
What is a web crawler?
A web crawler is a program that search engines use to discover and read web pages by following links from one page to the next. Googlebot is the crawler Google uses.
Why it matters
A page that no crawler reaches cannot be indexed, so it cannot rank. Crawlers find pages through links and through the XML sitemap.
AI companies run crawlers too. Their crawlers read pages so that AI assistants can use the content in answers.
How we handle it
We crawl a site with our own crawler, Screaming Frog, to see what a search engine sees. Server logs then show which pages the real crawlers request. The difference between the two lists tells us where crawling is wasted.
Related service: Technical SEO services
Full guide: Technical SEO checklist
Related terms
Keep reading
-
Crawl budget
Crawl budget is the amount of crawling a search engine is willing to spend on a website in a given period.
-
robots.txt
robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may request.
-
XML sitemap
An XML sitemap is a file that lists the pages of a website you want search engines to index.
-
Indexation
Indexation is the step in which a search engine stores a page in its database, called the index.
See how your site performs in search
Want this checked on your own site? Send us the address and we will tell you what we would look at first.