robots.txt and Sitemap Findings in a Site AI Audit Report

Short answer: The second step of a Site AI Audit check reads your robots.txt file and your XML sitemap, and it can raise four findings. A missing robots.txt is a notice. A robots.txt that blocks the whole site from search engines is critical. A missing XML sitemap is a warning. A sitemap that is not listed in robots.txt is a notice. They are all quick to fix, and the critical one can bring a whole website back into search results.

What the second step of the check does

On the home page the audit’s progress is described in stages, and the second is “robots.txt and sitemap”: what search engines are allowed to see, and which pages you ask them to index. Both files sit at fixed places on your domain. The robots.txt file is at the root, for example yoursite.com/robots.txt, and gives instructions to crawlers about which areas they may visit. The XML sitemap is a list of the pages that you want to appear in search, which helps crawlers find new pages quickly.

The audit reads both files the way a search engine would, and it respects them itself: its crawler, SiteAuditBot, follows the rules that apply to all crawlers and the ones addressed to it by name. That is why a robots.txt that blocks everything will also limit what the audit can see on your site. If you want more on that behaviour, the article about SiteAuditBot explains it.

Finding 1: robots.txt is missing

This is a notice. Without a robots.txt file, search engines assume that they may crawl everything, which is fine for a small site, but they also lose a convenient place to find your sitemap. The audit’s advice is to add a robots.txt file with a Sitemap line. A simple file for a typical small site can be only a few lines: allow everything, and point to the sitemap. If your CMS has an SEO plugin, it often creates the file for you, so check its settings before writing one by hand. The guide to robots.txt for beginners has safe examples.

Finding 2: robots.txt blocks the whole site

This is the most serious finding in this group, and it is critical. It appears when the file contains “Disallow: /” for all crawlers, which tells search engines not to crawl any page. The explanation in the report is blunt: Google will not crawl your pages, so they disappear from search. The advice is to remove the rule unless the site really should be hidden.

How does this happen? Usually, by accident. A developer blocks a staging copy during a redesign and the setting is copied to the live site at launch. Some CMS settings, such as the WordPress option to discourage search engines from indexing the site, have a similar effect. A site can run for months with this setting and nobody notices until traffic collapses. If you see this finding, treat it as urgent: remove the rule, check your CMS settings and then ask Google to re-crawl your site with the tools in Search Console.

Finding 3: no XML sitemap found

This is a warning. A sitemap helps search engines discover every page quickly, especially new ones. Without it, crawlers must find pages by following links, which works for a small, well-linked site and fails for pages that are buried. The audit’s advice is to generate a sitemap.xml, which most CMSs and SEO plugins can do automatically, and to add it to robots.txt.

If your site is large, one file may not be enough. A sitemap can hold up to 50,000 addresses, and bigger sites split their list into several files grouped by a sitemap index, as the article on sitemap index files explains. If you are unsure how to create or submit a sitemap, the XML sitemap guide covers it step by step.

Finding 4: the sitemap is not listed in robots.txt

When a sitemap exists but robots.txt does not mention it, the audit raises a notice. Listing it in robots.txt lets every search engine find it, not only Google, which you may have informed through Search Console. The fix is a single line in robots.txt: “Sitemap:” followed by the full address of the sitemap. The report even tells you the address to use, because it found the sitemap itself.

How to fix all four in ten minutes

Find out what you have. Open yoursite.com/robots.txt and yoursite.com/sitemap.xml in a browser. Common alternatives for the sitemap are sitemap_index.xml and wp-sitemap.xml.
Remove any rule that blocks everything. Look for “Disallow: /” under “User-agent: *”.
Create the sitemap if it is missing. Turn on the sitemap option of your SEO plugin or CMS.
Add the Sitemap line to robots.txt. Use the full address, including https://.
Run the check again. All four findings should be gone.

Be careful with robots.txt edits. A single wrong line can block the whole site, which is exactly what the critical finding warns about. Keep a copy of the old file before you change it, and read the new one carefully.

robots.txt is not a security tool

One misunderstanding comes up often: robots.txt does not hide anything. It is a polite request that well-behaved crawlers follow, and it is publicly readable. Listing a private folder in it can even point attackers to the folder, as the article on robots.txt revealing hidden paths explains. Protect private areas with a login or server rules, and use robots.txt only to guide search engines. To keep a page out of search results, a noindex tag is the right tool, and the audit has a separate finding for pages hidden that way; the comparison in robots.txt versus noindex explains the difference.

What the audit does when robots.txt blocks it

Because the audit follows your rules, a robots.txt that disallows the whole site also limits what the crawler may open. In that situation the critical finding is the most useful thing in the report, since it explains why the rest of the check has little to look at. After you fix the rule, run the check again to get the full picture, with titles, headings, links and every other finding for the pages that were hidden.

Check the rules for specific folders too

The finding about blocking the whole site is only the extreme case. A robots.txt can also block single folders by accident, such as the folder that holds your images, scripts or styles. Blocking those can prevent search engines from rendering your pages properly. Read the file line by line once, and ask of every Disallow rule: do I want search engines to stay out of this? If you cannot answer, remove the rule or ask the person who added it.

Staging sites and launches

The most expensive robots.txt mistakes happen around launches. A staging site should be blocked from search engines, and a live site should not be. The setting that does one often lives in the same place as the setting that undoes it, so a launch checklist should name it explicitly. Add three lines to yours: check robots.txt, check the sitemap address and check that the CMS option to discourage search engines is off. Then run the audit on the live address on launch day and again a week later, because some changes only show up after caches expire.

After the fix: how long until search results recover

Removing a blocking rule does not bring pages back instantly. Search engines have to crawl the site again and decide what to show, which can take days or weeks depending on the size of the site and how often it was visited before. You can speed it up by submitting the sitemap in Search Console and by linking to your important pages from the home page. Keep an eye on the number of indexed pages over the next weeks. The audit tells you that the door is open; the search engine decides when to walk in.

A worked example

A restaurant relaunches its website with a new design. Two weeks later, bookings from search drop sharply. The owner runs a Site AI Audit check, and the report shows a critical finding: robots.txt blocks the whole site from search engines. The agency’s staging setting had been left in place at launch. The owner asks the agency to remove the rule, checks that the file now allows crawling, and runs the audit again. The finding is gone, and a sitemap notice that was also there is cleared by adding the sitemap address to the file. The restaurant’s pages reappear in search within a few weeks, and the owner adds “check robots.txt” to the launch checklist for any future change.

To read your own robots.txt and sitemap the way a search engine does, check your website with Site AI Audit. The check needs a free account; the plans unlock the full report.

The bottom line

robots.txt and the sitemap are small files with big effects. Make sure the first does not block your site, that the second exists and that the first points to the second. The audit checks all of it in the second step of every run.

FAQ

Which of the four findings is the most serious?

The one saying robots.txt blocks the whole site from search engines. It is critical, because pages can disappear from search results.

Do I need a robots.txt file?

It is not required, but the audit flags a missing file as a notice because it is the usual place to point crawlers to your sitemap.

Where should the sitemap be?

Most sites use /sitemap.xml or /sitemap_index.xml. The audit finds it from robots.txt or from the common locations.

Does robots.txt keep a page out of Google?

Not reliably. It stops crawling, but a page can still appear in results if other sites link to it. Use noindex for that.

Will the audit obey my robots.txt?

Yes. Its crawler follows the rules for all crawlers and the ones addressed to SiteAuditBot.