Unless you often use Google Search Console, you might miss the Crawl Stats report. This handy feature, hidden in the Settings menu, uncovers a wealth of insights about how Google crawls your website.
This article will show you how to use the Google Search Console Crawl Stats report to optimize your crawl budget, streamline the crawl, and troubleshoot any issues.
When Should You Use the Crawl Stats Report?
You can always use the Crawl Stats report to check how Google indexes your site. However, it’s indubitably at its most useful for big websites, especially eCommerce stores that constantly update existing and launch new content.
For example, many new eCommerce businesses that aren’t familiar with SEO start by creating product pages and categories with no internal links. As the website grows, so does the content, with no rhyme or reason (nor robots.txt instructions). This makes it hard for Google to prioritize pages, schedule crawls, and identify the most important content.
Product pages stop getting picked up as fast, some go completely ignored, and there’s no predictability. And when SEO is one of your primary sources of traffic (and revenue), predictability matters.
The Crawl Stats report allows you to see what’s going on under the hood of your website SEO. And if there’s an issue, you can quickly troubleshoot it to ensure everything is in good shape.
Sample Questions for Your Crawl Stats Journey
A few of the questions you can answer with the Crawl Stats report include:
- Is Google spending my crawl budget on the pages that matter to my business?
- How much of my crawl budget is wasted on 404 or error pages?
- Do I have too many redirects?
- Is my internal link structure pointing to the most important pages?
- How often does Google crawl my existing content for updates?
- How fast does Google pick up my new content?
- Do I need to optimize my website?
And more! Of course, SiteGuru displays the critical information in a handy SEO to-do list based on weekly audits, but sometimes you’ll want to do it manually to get detailed insights.
Where Is the Crawl Stats Report in Google Search Console?
- Log into the Search Console
- Scroll down in the left-hand sidebar
- Select “Settings”
- Scroll to the “Crawling” section and the “Crawl stats” list item
- Click on “Open Report”
Critical Data in Your Crawl Stats Report
Total Crawl Requests
If you often update your website, Google will crawl it more often. However, you can check the frequency of the crawls in the main performance graph. You want to be seeing a relatively steady incline with no significant drops.
You’ll also see how much data Google is absorbing with each crawl. If your website is heavy on content, images, etc., it’s normal to see big Total download sizes.
However, that could slow down the bots, so optimize your assets as much as possible, mark the pages you don’t want to be crawled in the robots.txt file or with noindex tags, and monitor the Average response time.
Website Availability
One of the key things you can do to ensure an efficient crawl is to monitor your website’s uptime. If Google often tries to crawl your website and finds it unavailable, it’ll reduce your crawl budget (making it hard to get new and updated pages picked up fast).
Once you click on the “Host status” report, you’ll see information about your website availability.
Expand into each report if you see issues to determine when they occurred. If your host was down at some point, you might see the downtime in the chart if Google was crawling your website at the time.
Troubleshooting Host Status Issues in the Crawl Stats Report
| Host status action | Issue? |
|---|---|
| Robots.txt fetch | Make sure your robots.txt file is valid and accessible. |
| DNS resolution | Check your registrar to ensure you’ve set up the domain and host names correctly. |
| Server connectivity | Double-check that your server can handle the crawling requests. |
Analyze the Crawl Requests Breakdown
Next up, you will want to see how much of your crawl budget was spent on different status codes. If you see the majority in the 200 (OK) status code - you’re good! You want to optimize for that so your allotted crawl requests aren’t wasted on errors.
Troubleshooting Crawl Requests in the Search Console
| Status code | Issues |
|---|---|
| OK (200) | Nothing to do here! |
| Moved permanently (301) | Check if you meant to implement that redirect. Keep it if it looks good. Troubleshoot if there’s an accidental redirect. |
| Moved temporarily (302) | Check that the redirect is intentional. |
| Not modified (304) | Check your browser and server configuration for 304 issues. Consider clearing your cache, flushing the DNS, running an antivirus scan, and reviewing the redirect instructions in .htaccess. |
| Not found (404) | Remove links to pages that no longer function or set up redirects. |
| Server errors (5xx) | Check your server setup and uptime. Make sure Google can crawl all the pages you want it to crawl. |
Check Crawl Requests by File Type
In addition to crawling your HTML and textual content, Google will also crawl your images, JavaScript, CSS, and other website assets.
Ideally, you want to ensure it crawls HTML and images more than it crawls JS and CSS. Optimize and minify the code so your website doesn’t slow down.
Crawl Requests by Purpose in the Search Console Crawl Stats
Crawl requests by purpose will show you if Google mainly crawls your website to see updated content or discover new content. This depends on your strategy, so the stats here should reflect what you’re doing with the website.
Check the pages for the following:
- Priority. If Google crawls a handful of pages more often, it perceives them as more important.
- Irrelevant pages. Reduce your crawl budget by no-indexing the pages Google might be picking up (and that you don’t care about).
- Important pages not getting crawled. Add more internal links and check your sitemap to emphasize their importance.
Googlebot Type
Finally, the Googlebot Type crawl stats will show you which bot types Googlebot uses to crawl your website. This depends on your content type.
For example, you might see the AdBot spike when you add new Google Ads campaigns. Similarly, Google will fetch your CSS code through the Page Resource Load. If you add a Product Feed, you’ll see the Store Bot visit your site more often.
In short, ensure Google crawls your website with the right bot. If you’re in eCommerce, you’ll want to see the Store Bot. You’ll want to see the Smartphone and Desktop bots if you run a content website.
Fix Your Crawl (and the Rest Is a Breeze)
Without crawling, there’s no indexation. And without indexation, you can’t rank. So if you or SiteGuru have identified issues with your website, head to the Crawl Stats report. It’s the crucial first step on your SEO journey!
