A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which parts of the site they may and may not crawl. Google, Bing and most other well-behaved bots read it before they crawl anything else. It controls crawling, not indexing: a page you block can still show up in Google if other pages link to it.
Where does robots.txt live?
Always at the root of the host it applies to, and always named in lowercase:
https://www.example.com/robots.txt
You can open any site's file by adding /robots.txt to its domain. That also means anyone can read yours, so never use it to hide private pages.
Each host needs its own file. www.example.com, blog.example.com and shop.example.com each read only the robots.txt at their own root. A file in a subfolder, like example.com/pages/robots.txt, is ignored.
If you don't have a robots.txt file at all, crawlers assume they can crawl everything. For many small sites, that's fine.
What does a robots.txt file look like?
Here's a typical one:
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
In plain words: "Every crawler, please don't crawl the cart or the internal search results. Here's where the sitemap is." The file is made of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers.
Which rules can you use?
| Rule | What it does |
|---|---|
User-agent | Names the crawler the rules below it apply to. * means every crawler. |
Disallow | A path the crawler shouldn't crawl. An empty value (Disallow:) blocks nothing. |
Allow | A path the crawler may crawl, even inside a folder you disallowed. Google and Bing both support it. |
Sitemap | The full URL of your XML sitemap. You can list more than one. |
Crawl-delay | Seconds a crawler should wait between requests. Bing respects it. Google ignores it. |
These rules follow the Robots Exclusion Protocol, which became an official internet standard (RFC 9309) in 2022.
Anything else you may see in old guides, like a Noindex: line, does nothing. Google stopped supporting it in 2019.
Common examples
Allow everything
User-agent: *
Disallow:
The empty Disallow means "block nothing." This has the same effect as having no file.
Block the whole site
User-agent: *
Disallow: /
Use this on a staging site, never on your live site. A stray Disallow: / left over after a launch is one of the most common ways sites drop out of Google.
Block one folder
User-agent: *
Disallow: /admin/
The trailing slash matters. Disallow: /admin would also block /administrator and /admin-tips, because a rule matches every path that starts with it.
Block one folder, but allow one page inside it
User-agent: *
Disallow: /downloads/
Allow: /downloads/price-list.pdf
When two rules match the same URL, Google uses the most specific one (the longest path). Here the Allow line wins for the price list.
Block one crawler only
User-agent: Bingbot
Disallow: /experiments/
Only Bing's crawler skips /experiments/. Everyone else can still crawl it.
How do wildcards work?
Two characters let you match patterns instead of exact paths:
*matches any sequence of characters.Disallow: /*?sort=blocks every URL with a sort parameter.$marks the end of the URL.Disallow: /*.pdf$blocks URLs that end in .pdf.
Paths are case-sensitive. Disallow: /Photos/ doesn't block /photos/.
What should you block, and what shouldn't you?
Block pages that waste a crawler's time and have no reason to rank:
- Internal search result pages
- Cart, checkout and account pages
- Filter and sort combinations that create thousands of near-identical URLs (see crawl budget)
Don't block:
- Pages you want out of Google. Use a noindex tag instead, and leave the page crawlable so Google can see the tag. If you block it in robots.txt, Google never reads the noindex, and the URL can stay indexed. We explain this in detail in Indexed, though blocked by robots.txt.
- CSS, JavaScript and images your pages need. Google renders your page like a browser does. If it can't load those files, it may not see your content or know whether the page works on mobile.
- Duplicate pages you've set a canonical on. Google has to crawl a page to see its canonical tag.
Can you block AI crawlers?
Yes. AI companies run their own crawlers, and most of them follow robots.txt. For example:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
GPTBot is OpenAI's crawler, ClaudeBot is Anthropic's and PerplexityBot is Perplexity's. Google uses a separate token, Google-Extended, to let you opt out of your content being used for its Gemini models without affecting Google Search.
Think before you block them. If AI crawlers can't read your site, AI answers can't quote or link to it. Our guide on ranking in AI search covers the trade-off.
How do you check your robots.txt?
- Open yourdomain.com/robots.txt in a browser and read it. Look for a stray
Disallow: /first. - In Google Search Console, go to Settings > robots.txt. The robots.txt report shows the version Google last fetched and any lines it couldn't parse.
- To check one URL, paste it into URL Inspection and click Test Live URL. Google tells you if robots.txt blocks it.
Google caches your file for up to 24 hours, so a change can take a day to be picked up. If you need Google to see it sooner, use Request a recrawl in the robots.txt report.
Google's own robots.txt documentation is the reference if you need the full rules.
What to do next
Open your robots.txt now and check it against the "don't block" list above. Then look at which of your pages are actually indexed. The indexation report guide shows how to line up what your site tells Google with what Google actually does.
