#robots.txt
The robots.txt file sits at the root of a domain and tells crawlers which paths they may request. It controls crawling, not indexing, which is a common source of confusion. A single wrong line can block an entire website from search engines. The file is public, so it should never be used to hide sensitive content. Articles under this tag explain the syntax, typical mistakes and safe ways to test changes.
15 de nov. de 2026 · 8 min de leiturarobots.txt and Sitemap Findings in a Site AI Audit Report
Is robots.txt missing, or blocking your whole site? Is there no sitemap? Learn how Site AI Audit reports these four findings and how to…
11 de nov. de 2026 · 8 min de leituraSiteAuditBot: What the Site AI Audit Crawler Does on Your Site
Wondering what SiteAuditBot is? See how the Site AI Audit crawler checks your website, how gently it behaves, what it reads and how robots.txt…
8 de nov. de 2026 · 8 min de leiturallms.txt Explained: Does Your Website Need One?
llms.txt is a proposed plain-text file that points AI assistants to your best pages. Learn what it contains, what it can and cannot do…
2 de nov. de 2026 · 8 min de leituraWhen Google Cannot Fetch robots.txt: What Happens Next
If Google cannot fetch your robots.txt file, crawling may slow or pause. Learn what each error means, how long it matters and how to…
31 de out. de 2026 · 8 min de leituraRobots.txt Can Reveal Hidden Paths: What Not to List
Robots.txt is public, so every disallowed path is a hint for attackers. Learn what to keep out of the file and how to protect…
9 de out. de 2026 · 8 min de leituraIndexed, Though Blocked by robots.txt: What It Means and Fixes
Search Console says a page is indexed, though blocked by robots.txt. Learn why Google indexes pages it cannot crawl and how to fix it…
1 de out. de 2026 · 8 min de leituraStaging Site Indexed by Google? How to Fix and Prevent It
Found your staging or dev site in Google results? Here is how to remove it quickly, why robots.txt alone fails, and how to keep…
29 de ago. de 2026 · 8 min de leituraRobots.txt vs Noindex: Which One Should You Use?
Robots.txt controls crawling, noindex controls indexing. Learn the difference, which to use for common page types and why combining them backfires.
10 de ago. de 2026 · 8 min de leituraRobots.txt Explained: A Beginner’s Guide With Safe Examples
What robots.txt does and does not do, how its rules work, safe examples for WordPress and shops, and the mistakes that can block a…