ENEnglish
LTLietuviškai
DEDeutsch
FRFrançais
RUРусский
ESEspañol
PLPolski
ITItaliano
UKУкраїнська
CSČeština
PTPortuguês
JA日本語
LVLatviešu valoda
ETEesti
HUMagyar
IDBahasa Indonesia
TRTürkçe
HIहिन्दी
ELΕλληνικά
ARالعربية
THไทย
VITiếng Việt
RORomână
MSBahasa Melayu
BNবাংলা#robots.txt
The robots.txt file sits at the root of a domain and tells crawlers which paths they may request. It controls crawling, not indexing, which is a common source of confusion. A single wrong line can block an entire website from search engines. The file is public, so it should never be used to hide sensitive content. Articles under this tag explain the syntax, typical mistakes and safe ways to test changes.
2026/10/01 · 8分で読めますStaging Site Indexed by Google? How to Fix and Prevent It
Found your staging or dev site in Google results? Here is how to remove it quickly, why robots.txt alone fails, and how to keep…
2026/08/29 · 8分で読めますRobots.txt vs Noindex: Which One Should You Use?
Robots.txt controls crawling, noindex controls indexing. Learn the difference, which to use for common page types and why combining them backfires.
2026/08/10 · 8分で読めますRobots.txt Explained: A Beginner’s Guide With Safe Examples
What robots.txt does and does not do, how its rules work, safe examples for WordPress and shops, and the mistakes that can block a…