Glossary
What is crawl budget?
Crawl budget is the amount of crawling a search engine is willing to spend on a website in a given period. It depends on how fast the server responds and on how valuable the search engine thinks the pages are.
Why it matters
On a small site, crawl budget is never a problem. On a site with tens of thousands of URLs, search engines may spend most of their visits on filters and duplicates, and reach important pages late or not at all.
The symptom is slow indexation: new pages take weeks to appear in results, and updates to old pages are not picked up.
How we handle it
Server logs show which URLs search engines request. We remove low-value URLs from the crawl with canonical tags, noindex rules and robots.txt, and keep the sitemap limited to pages that should rank. In one project this meant cutting 66,517 indexed pages to 324, as shown in our EdTech case study.
Related service: Technical SEO services
Related terms
Keep reading
-
Noindex
Noindex is an instruction that tells search engines not to show a page in their results.
-
Canonical tag
A canonical tag is a line in the HTML of a page that tells search engines which URL is the main version when several URLs show the same or very similar content.
-
Indexation
Indexation is the step in which a search engine stores a page in its database, called the index.
See how your site performs in search
Tell us what is happening with your search traffic. You get a straight answer on where we would start.