SEO glossary
robots.txt
robots.txt is a text file at your site root that tells crawlers which URLs they may or may not request.
In plain English
It controls crawling, not indexing — a URL blocked in robots.txt can still be indexed if linked elsewhere. Use it to keep bots out of infinite spaces and private areas, and to point to your sitemap.
Why it matters for SEO
A single overly broad Disallow line can accidentally block your whole site from being crawled.
Related terms
Frequently asked questions
- What is robots.txt?
- robots.txt is a text file at your site root that tells crawlers which URLs they may or may not request.
- Why does robots.txt matter for SEO?
- A single overly broad Disallow line can accidentally block your whole site from being crawled.
- How does robots.txt work?
- It controls crawling, not indexing — a URL blocked in robots.txt can still be indexed if linked elsewhere. Use it to keep bots out of infinite spaces and private areas, and to point to your sitemap.
Related
Last updated