Robots.txt
A robots.txt file is a plain-text file in a website’s root directory that tells crawlers which URLs or sections they may or may not crawl.
Key takeaways
- Robots.txt controls crawling, not indexing.
- Place it at the root of your site.
- Use it to manage crawl budget, not to hide content.
- Always test changes with Google Search Console.
- Do not block essential resources like CSS or JS.
For SEO, robots.txt is mainly a crawl-control tool: it helps direct search-engine bots away from low-value or resource-heavy URLs so they can spend more attention on important pages.
Example
A simple robots.txt that blocks all crawlers from the entire site:
User-agent: *
Disallow: /A more practical example that allows all crawlers but blocks access to the admin area:
User-agent: *
Disallow: /admin/When should I use this?
- To manage crawl budget – Block low-value pages (e.g., search results, paginated archives) so crawlers focus on important content.
- To prevent crawling of sensitive areas – Like admin panels, staging sites, or duplicate content.
- To block large files – Such as PDFs or images that consume crawl resources without SEO value.
What it is not
- Not a security measure – It only asks compliant bots to stay out; malicious bots and users can still access blocked URLs.
- Not a way to hide pages from search results – Google may still index a URL if it is linked from elsewhere, even if blocked from crawling.
- Not a replacement for noindex – Use noindex meta tag or HTTP header to prevent indexing.
- Not a guarantee – Some crawlers ignore robots.txt entirely.
Quick start
- Create a plain text file named
robots.txt. - Add rules using
User-agentandDisallow(orAllow) directives. - Place the file at the root of your domain (e.g.,
https://www.example.com/robots.txt). - Test it using Google Search Console's robots.txt Tester.
- Monitor crawl stats in Search Console to see if the changes affect crawl behavior.
Common mistakes
- Blocking important resources – CSS, JS, or images that Google needs to render pages correctly.
- Using robots.txt to hide pages from search results – Instead, use noindex or password protection.
- Wrong file location – Must be at the root, not in a subfolder.
- Assuming it is a security tool – It only deters well-behaved bots, not malicious ones.
- Blocking the sitemap – Ensure your sitemap URL is not disallowed.
Next step
FAQ
Can robots.txt keep a page out of Google search results?
No. Robots.txt only prevents crawling, not indexing. If a page is linked from elsewhere, Google may still index it without crawling. Use noindex or password protection to block indexing.
Where should I place my robots.txt file?
At the root of your domain, e.g., https://www.example.com/robots.txt. It must be accessible to crawlers.
Related topics
Sources
- Google Search Central — robots.txt Introduction and Guide — Most authoritative source for how Google interprets and uses robots.txt.
- Google Search Console Help — robots.txt — Clear guidance on what robots.txt can and cannot do for crawling and indexing.
- Google for Developers — Create and Submit a robots.txt File — Explains file location, rule structure, and basic implementation.
- MDN Web Docs — Robots.txt glossary — Concise technical definition and placement basics.
Reviewed by Lucía Marín, Founding editor.