For most business websites, crawl budget does not matter. Google’s own guide targets sites with 1 million or more unique pages changing weekly, sites with 10,000 or more pages changing daily, and sites with many URLs stuck at “Discovered, currently not indexed.” If new pages get crawled within days, look at content quality and indexing instead.
Few phrases in SEO are used to sell more unnecessary work than “crawl budget.” It sounds technical, it cannot be seen from the outside, and a report that says your crawl budget is being wasted is hard for a business owner to challenge.
Google has, in fact, told us who needs to think about it, and it is almost none of the sites that get audited for it.
1. What crawl budget is
Crawl budget is the set of URLs on your site that Googlebot can and wants to crawl. Google describes two parts. Crawl capacity is how much crawling your server can handle without being overloaded; it rises when your site responds quickly and consistently and falls when it slows or returns server errors. Crawl demand is how much Google wants to crawl, driven by how many URLs it knows about, how popular they are, and how stale they look.
One more point in Google’s documentation gets lost in most SEO commentary: crawling is not indexing. A page that has been crawled is not automatically indexed, because each one is evaluated and consolidated first. That distinction is the key to everything below.
2. Who actually needs to worry
Google’s guide is explicit about its audience. It is an advanced guide for large sites with more than a million unique pages whose content changes about weekly, for medium or larger sites with more than 10,000 unique pages whose content changes daily, and for sites where a large portion of URLs is classed in Search Console as Discovered, currently not indexed. Google describes these as rough estimates rather than hard cutoffs.
It also says the guide is not needed if your site does not have many rapidly changing pages, or if your pages seem to be crawled the same day they are published. For everyone else, it advises keeping the sitemap up to date and checking the Page indexing report.
A 150 page plumbing site, a 400 page clinic site and a 2,000 page local directory are all far below the lines. Selling “crawl budget optimization” to any of them is selling a solution to a problem they do not have. We would say the same about our own audits: if one of our reports lists crawl budget as a priority for a site this size, ask us to justify it.
3. The misdiagnosis that wastes the most money
The usual story goes like this. An owner publishes a page, Google does not index it, and someone blames crawl budget. In most small sites the cause is something else. Google sometimes crawls a page and decides, for now, that it does not add enough to be indexed. That is a content problem, and no amount of crawl optimization fixes it.
Note also that Google says new pages typically take several days at minimum to be noticed, and most sites should not expect same-day crawling unless they are time-sensitive, such as news. A page that is not indexed after two days is not evidence of anything.
When a page is stuck at Crawled, currently not indexed, ask whether it duplicates another page, whether it is thin, and whether anything links to it. Those three questions solve most cases.
4. When it does matter
There are real cases. Large ecommerce sites with filters and sorting options can generate millions of URL combinations from a catalog of a few thousand products. Sites that auto-generate pages in bulk can create inventory that dwarfs the real content. A compromised site can suddenly host thousands of injected URLs, as we describe in our guide to how a hacked website destroys your rankings. And a slow or erroring server lowers capacity directly, because Google backs off when responses slow down or return 5xx or 429 errors.
Site moves are a special case. Google notes that a move can temporarily raise crawl demand, which is why a server that coped before a migration can struggle after it. If you are planning one, read website migration without losing rankings first.
5. How to check your own site in fifteen minutes
You do not need a log analysis tool to find out whether you have a crawl problem. Search Console gives you enough to decide.
- Compare submitted and indexed. In the Sitemaps report, note how many URLs you submitted. In the Page indexing report, note how many are indexed. A large gap on a site of a few hundred pages is nearly always a quality or duplication issue.
- Read the reasons, not the totals. Open each “not indexed” reason and look at example URLs. Ten pages with the same reason usually share the same cause.
- Open the Crawl stats report. It sits under Settings in Search Console. Look at the trend in crawl requests and average response time, and at the host status section for availability problems. Rising response times alongside falling crawl requests is the pattern Google describes when capacity drops.
- Test a new page. Publish a page, request indexing through URL Inspection, and note how long it takes. If it appears within a few days, your crawl is healthy by Google’s own yardstick.
If none of those four steps shows a problem, you have your answer. Spend the time you would have spent on crawl budget on the content the pages are missing.
6. What to do on a normal-sized site instead
The unglamorous list is also the effective one. Keep your XML sitemap current and limited to pages you want indexed. Check the Page indexing report monthly and read the reasons, not just the counts. Merge or remove thin and duplicate pages. Fix soft 404s, which Google says continue to be crawled and waste budget. Link to your important pages from other important pages. And keep the server responsive: a fast, stable site raises crawl capacity as a side effect, and the fixes overlap with our Core Web Vitals guide.
| Local service business | Large online catalog | |
|---|---|---|
| Typical size | About 20 to 500 pages | Tens of thousands to millions of URLs |
| Real risk | Thin, duplicate or unlinked pages not being indexed | Filters and parameters creating huge URL sets |
| Right response | Content, internal links, sitemap hygiene | Crawl control, canonicalization, log analysis |
| Crawl budget work worth paying for? | Almost never | Yes |
If you want to know which side of that table you are on, a technical review can usually settle it quickly; the ongoing work belongs in a monthly plan, and our SEO packages and pricing page shows what that looks like.
Know whether it is a crawl problem or a content problem
We read your Page indexing report line by line and tell you plainly whether anything on your site is worth fixing, including when the answer is nothing.
Request a technical reviewFrequently asked questions
What is crawl budget?
Google defines crawl budget as the set of URLs it can and wants to crawl on your site. It depends on crawl capacity, how much your server can handle, and crawl demand, how much Google wants to crawl based on the number, popularity and freshness of your URLs.
Does my small business website need to worry about crawl budget?
Almost certainly not. Google’s guide is aimed at sites with more than a million unique pages changing about weekly, sites with more than 10,000 pages changing daily, and sites with a large share of URLs marked Discovered, currently not indexed. A site with a few hundred pages is nowhere near those lines.
Why are my pages crawled but not indexed?
Because crawling is not indexing. Google evaluates each crawled page and may decide not to index it for now. The usual causes are duplication, thin content or lack of internal links, which are content problems rather than crawl budget problems.
How long should it take Google to index a new page?
Google says that for most sites, new pages take several days at minimum to be noticed, and most sites should not expect same-day crawling unless they are time-sensitive, such as news sites. A page not indexed after two days is not evidence of a problem.
Can a slow website reduce how much Google crawls?
Yes. Google says the crawl capacity limit rises when a site responds consistently and quickly and falls when it slows down or returns 5xx or 429 errors. A slow server can reduce crawling even when Google would like to crawl more.
Do soft 404s waste crawl budget?
Google says soft 404 pages continue to be crawled and waste budget, and recommends finding them in the Page indexing report and fixing them. On a small site the more important reason to fix them is that they signal empty or broken pages to users.
Sources
- Google Search Central: “Optimize your crawl budget” (large site owner’s guide). Intended audience and rough thresholds, definitions of crawl capacity limit and crawl demand, the statement that crawled pages are not automatically indexed, the soft 404 guidance, and the note that site moves can temporarily raise demand.
- Google Search Central, large site guide: for most sites new pages take several days at minimum to be noticed, and same-day crawling is not to be expected outside time-sensitive sites.
- Google Search Console Help: Page indexing report, status definitions for “Discovered, currently not indexed,” “Crawled, currently not indexed,” duplicate and soft 404 statuses.
Note: the typical page count for a local service business site (roughly 20 to 500 pages) is our observation across client sites, not a published statistic. Google describes its crawl budget thresholds as rough estimates and not hard cutoffs.
Reference: Read the original research or documentation.

Muhammad Asjad Khan is a seasoned digital marketer and the Chief Operating Officer at Digital Age, where he leads strategy, innovation, and client success. With years of experience in SEO, content marketing, and performance-driven digital campaigns, Asjad is passionate about helping brands grow their online presence and achieve measurable results. When he’s not optimizing websites or scaling marketing strategies, he’s exploring the latest trends in tech and digital media