Skip to content
SEO Catalyst

Glossary

What is robots.txt?

robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may request. It controls crawling, which means it does not remove a page from search results.

Why it matters

A single wrong line can block a whole site from being crawled. This happens most often when a rule from the staging site is carried over at launch.

Blocking a page here does not hide it. If other sites link to a blocked page, Google can still show its address in results. To keep a page out of results, use a noindex tag and leave the page crawlable.

How we handle it

We check the file before and after every release. We use it to keep crawlers away from filters and internal search results, and we add the address of the XML sitemap to it.

Related service: Technical SEO services
Full guide: Technical SEO checklist

Related terms

Keep reading

All glossary terms

See how your site performs in search

Want this checked on your own site? Send us the address and we will tell you what we would look at first.

Free SEO audit Book a call