Glossary
What is robots.txt?
robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may request. It controls crawling, which means it does not remove a page from search results.
Why it matters
A single wrong line can block a whole site from being crawled. This happens most often when a rule from the staging site is carried over at launch.
Blocking a page here does not hide it. If other sites link to a blocked page, Google can still show its address in results. To keep a page out of results, use a noindex tag and leave the page crawlable.
How we handle it
We check the file before and after every release. We use it to keep crawlers away from filters and internal search results, and we add the address of the XML sitemap to it.
Related service: Technical SEO services
Full guide: Technical SEO checklist
Related terms
Keep reading
-
Web crawler
A web crawler is a program that search engines use to discover and read web pages by following links from one page to the next.
-
Noindex
Noindex is an instruction that tells search engines not to show a page in their results.
-
Crawl budget
Crawl budget is the amount of crawling a search engine is willing to spend on a website in a given period.
-
XML sitemap
An XML sitemap is a file that lists the pages of a website you want search engines to index.
See how your site performs in search
Want this checked on your own site? Send us the address and we will tell you what we would look at first.